Re: [openstack-dev] [FaaS] Function as a service in OpenStack

[Dmitri Zimine]

 Hi all,

please let me re-surface the discussion,
and propose StackStorm as [a starting point for] a Serverless framework for
OpenStack.

StackStorm [0] is an open source event-driven automation platform, built
around Mistral and adds “before” and “after” - (triggers on events, and
rules that maps triggers to actions) and after (easy to add actions with
auto-generated API) [1].
It is built for devops automation and often used to automate OpenStack [2].
Recently, we saw it used as a DIY serverless framework, as it has the parts
of Serverless stack discussed here: “Lambda”, API Gateway, event sources
(not just timers and web hooks but any other events as triggers), and
workflows/Step Functions with Mistral - as workflows become recognizable
part of serverless.

Comparing with OpenWhisk, StackStorm brings the same concepts and
functionality, but does it on OpenStack technology stack: Python, RabbitMQ,
Pecan, eventlets, Oslo config & utils, and a fair amount of same 3rd party
dependencies.

StackStorm is mature (3 years old), heavily used (~3,000 installations /
month), have rich set of existing integrations [3]. StackStorm team has
been a part of OpenStack community and Mistral contributors from the outset.

I hate that this is coming out as a plug, but I’d hate it even more if
there is a match, and we miss it.
Let’s decide on merits. The detailed discussion takes more writing…

How about we use the Boston summit to open the conversation? Here is what
we can do:

1) My talk “Serverless on OpenStack with StackStorm and Mistral” Wed May10
9am can serve as an introduction: https://goo.gl/2ZUJFK

2) Let’s make time at the summit with those who’re interested AND in
Boston, do in-depth technical, architecture, and merits discussion, and
share the findings here.

3) If the community believes the idea worth further consideration, we take
it from there.

If you’re interested to participate, reply here and/or contact me directly.

[0] StackStorm: https://github.com/StackStorm/st2

[1] Slides from OpenStack Barcelona where I explained the relations between
Mistral and StackStorm to Mistral community.
https://www.slideshare.net/DmitriZimine/mistral-and-stackstorm

[2] E.g. Cybera, or Symantec:
 -
https://www.openstack.org/videos/video/sleep-better-at-night-openstack-cloud-auto-healing
 -
https://www.mirantis.com/blog/auto-remediation-making-an-openstack-cloud-self-healing/


[3] Integration points (aka packs) https://exchange.stackstorm.org/



On Sat, May 6, 2017 at 12:25 PM, Dmitri Zimine <dzim...@brocade.com> wrote:

>
>
> From: Lingxian Kong <anlin.k...@gmail.com>
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: Wednesday, November 2, 2016 at 6:20 PM
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [FaaS] Function as a service in OpenStack
>
> On Thu, Nov 3, 2016 at 10:44 AM, Zane Bitter <zbit...@redhat.com> wrote:
>
>> This is a really interesting space. There seems to be two main use cases
>> for Lambda that are probably worth talking about separately:
>>
>> The first is for just Lambda alone. You can use it to provide some glue
>> logic between the other AWS services, so you can trigger off various events
>> (e.g. S3 notifications) and write a little bit of conditioning logic that
>> transforms the data and dispatches it to other services (e.g. DynamoDB).
>> This one is particularly interesting to me, and in fact we can support
>> parts of this in OpenStack already[1] because Mistral's functionality is
>> equivalent to something like SWS + parts of Lambda. (Specifically, Mistral
>> can do the data dispatch easily enough, but any data transformation has to
>> be done in YAQL, which is a pretty high bar compared to just writing some
>> code in a language of your choosing.)
>>
>
> ​There is still one thing missing in Mistral​ (maybe it should not be).
> After receieving events from Aodh or Zaqar, what if user just wants to
> trigger some scripts under his/her management, rather than just invoking
> openstack services api? Although actions are pluggable in Mistral, but in
> this case it's definitely not an easy thing as just writing an customized
> action, unless Mistral could include such capatility in its scope which I
> think it too heavy for Mistral to mange such things by itself. So, FaaS
> will be the right answer in this case, and it will also be add-on part to
> empower Mistral to do more things.
>
>
>>
>> The second one is Lambda + the API Gateway, which allows you to have web
>> requests act as triggers, so that you can effectively treat it as a PaaS
>> and build an entire web app by stringing together Lam

Re: [openstack-dev] [mistral] Who's interested in attending PTG?

[Dmitri Zimine]

StackStorm is game for PGT: Winson will likely be there for Mistral F2F:
from what we see now it’s quite a few things where F2F get-together can
help  & accelerate.

So if Renat you’re in, we have 3 companies.


Cheers, Dmitri.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mistral] How to call 3rd-party tools(such as Ansible) in Mistral

[Dmitri Zimine]

Tony, 

You can also connect Mistral with Ansible via StackStorm and Ansible 
integration pack: 
1) StackStorm http://docs.stackstorm.com
2) https://github.com/StackStorm/st2contrib/tree/master/packs/ansible

StackStorm is open-source Apache 2.0 “wrapper” around Mistral workflow service 
that integrates 3rd party tools.

Disclosure: I work for StackStorm, I think it’s appropriate to pitch here an 
open source project that addresses the problem.

DZ.


On Oct 15, 2015, at 11:35 PM, WANG, Ming Hao (Tony T) 
 wrote:

> Dear Mistral developers and testers,
>  
> We have developed some Ansible playbooks for operation automation, and we are 
> investigating if we can change the automation engine from Ansible to Mistral 
> since Mistral has powerful workflow control.
> Could you please help to provide how to let Mistral call Ansible playbook or 
> Ansible module?
> My understanding is to use ssh std_action to let Mistral access Ansible 
> server to call Ansible playbook/modules since Mistral ad-hoc action must base 
> on an existing system action.
>  
> Another question is:
> If we install Mistral and Ansible on a same server, is it possible to 
> simplify it?
>  
> Thanks in advance,
> Tony
>  
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mistral][yaql] Addressing task result using YAQL function

[Dmitri Zimine]

Yes meant to ask for consistency of referencing to task results. So it’s 
task(task_name) regardless of where. 

One use case in favor of this is tooling: I refactor workflow with an automated 
tool which wants to automatically rename the task name EVERYWHERE. You guys 
know well by now that renaming the task is a source of too many frustrating 
errors :)

What other think? 

DZ. 

On Sep 3, 2015, at 4:23 AM, Renat Akhmerov <rakhme...@mirantis.com> wrote:

> 
>> On 02 Sep 2015, at 21:01, Dmitri Zimine <dzim...@stackstorm.com> wrote:
>> 
>> Agree, 
>> 
>> with one detail: make it explicit -  task(task_name). 
> 
> So do you suggest we just replace res() with task() and it looks like
> 
> task() - get task result when we are in “publish”
> task(task_name) - get task result from anywhere
> 
> ?
> 
> Is that correct you mean we must always specify a task name? The reason I’d 
> like to have a simplified form (w/o task name) is that I see a lot of 
> workflows that we have to repeat task name in publish so that it just look 
> too verbose to me. Especially in case of very long task name.
> 
> Consider something like this:
> 
> tasks:
>   get_volumes_by_names:
> with-items: name in <% $.vol_names %>
> workflow: get_volume_by_name name=<% $.name %>
> publish:
>   volumes: <% $.get_volumes_by_names %>
> 
> So in publish we have to repeat a task name, there’s no other way now. I’d 
> like to soften this requirement, but if you still want to use task names 
> you’ll be able to.
> 
> 
>> res - we often see folks confused by result of what (action, task, workflow) 
>> although we cleaned up our lingo: action-output, task-result, 
>> workflow-output…. but still worth being explicit.
>> 
>> And full result is being thought as the root context $.
>> 
>> Publishing to global context may be ok for now, IMO.
> 
> Not sure what you meant by "Publishing to global context”. Can you clarify 
> please?
> 
> 
> Renat Akhmerov
> @ Mirantis Inc.
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mistral][yaql] Addressing task result using YAQL function

[Dmitri Zimine]

Agree, 

with one detail: make it explicit -  task(task_name). 

res - we often see folks confused by result of what (action, task, workflow) 
although we cleaned up our lingo: action-output, task-result, workflow-output…. 
but still worth being explicit.

And full result is being thought as the root context $.

Publishing to global context may be ok for now, IMO.

DZ>

On Sep 2, 2015, at 4:16 AM, Renat Akhmerov  wrote:

> Hi,
> 
> I’d like to propose a few changes after transition to yaql 1.0:
> 
> We already moved from using “$.__execution” in DSL to "execution()” and from 
> “$.__env” to “env()” where “execution()” and “env()" are registered yaql 
> functions. We had to do it because double underscored are prohibited in the 
> new yaql.
> 
> 
> In the spirit of these changes I’m proposing a similar change for addressing 
> task result in Mistral DSL.
> 
> Currently we have the syntax “$.task_name” that we can use in yaql 
> expressions to refer to corresponding task result. The current implementation 
> is of that is kind of tricky and, IMO, conceptually wrong because referencing 
> this kind of key leads to DB read operation that’s requried to fetch task 
> result (it may be big as megabytes so it can’t be stored in workflow context 
> all the time. In other words, we create a dictionary with side effect and 
> change the initial semantics of dictionary. Along with mentioned trickiness 
> of this approach it’s not convenient, for example, to debug the code because 
> we can accidentally cause a DB operation. 
> 
> So the solution I’m proposing is to have an explicit yaql function called 
> “res” or “result” to extract task result.
> 
> res() - extracts the result of the current task, i.e. in “publish” clause.
> res(‘task_name’) - extracts the result of the task with the specified name. 
> Can also be used in “publish” clause, if needed
> 
> This approach seems more flexible (cause we can add any other functions w/o 
> having to make significant changes in WF engine) and expressive because user 
> won’t confuse $.task_name with addressing a regular workflow context variable.
> 
> Of course, this to some extent breaks backwards compatibility. But we already 
> changed yaql version which was necessary anyway so it seems like a good time 
> to do it.
> 
> I’d very much like to have your input on this.
> 
> Renat Akhmerov
> @ Mirantis Inc.
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral][Murano] What's the latest/greatest on YAQL?

[Dmitri Zimine]

With YAQL migration on the way [1] 
I suggest we include a conversion script to the blueprint [2] and call BP done 
when conversion in place. 

DZ 


[1] https://review.openstack.org/#/c/206944/ 
[2] https://blueprints.launchpad.net/mistral/+spec/yaql-v1-0
On Jun 30, 2015, at 3:43 PM, Thomas Goirand z...@debian.org wrote:

 On 06/30/2015 07:09 AM, Dmitri Zimine wrote:
 Thanks Stan, 
 
 This release is few more months. How soon?  Are you planning to support
 0.2 in the meantime?
 
 We are really pressed on this transition to 1.0. 
 For instance. Number 1 user error while dealing with YAQL is using ==
 instead of =. 
 We had this discussion and you and Alex and you suggested it’s easy to
 redefine. 
 
 But… … The short is we need to move to 1.0 to do it. Because from what I
 figured so far, in 0.2 I can’t naively extend the an operator, as it
 won’t parse. I did make it work on 0.2 but this required a hack in the
 YAQL library itself, need to add ‘==‘ to both lexer.py and parser.py. At
 least what I figured. 
 
 I see the tokens are already generalized in 1.0. It doesn’t seem to
 worth backporting it to 0.2, but if this is a way, let us know.
 
 Mistral is betting on YAQL, please help.
 
 DZ. 
 
 FYI, I'm also looking forward switching to Yaql 1.0 on the packaging
 level, because 0.2.6 has lost support for Py3 (I added some Py3 patches
 in Debian for 0.2.4, but 0.2.6 completely broke that...).
 
 Cheers,
 
 Thomas Goirand (zigo)
 
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mistral] [murano] An online YAQL evaluator

[Dmitri Zimine]

Awesome! Really.

Thank you folks for doing this. 

I am so much looking forward to moving it to 1.0 with more built-in functions 
and more power to extend it...

Note that Mistral has a few extensions, like `str`, `len`, which are not in the 
scope of evaluator.

DZ 


On Aug 2, 2015, at 12:44 PM, Stan Lagun sla...@mirantis.com wrote:

 Guys, this is awesome!!!
 
 Happy to see yaql gets attention. One more initiative that you may find 
 interesting is https://review.openstack.org/#/c/159905/
 This is an attempt to port yaql 1.0 from Python to JS so that the same can be 
 done in browser
 
 Sincerely yours,
 Stan Lagun
 Principal Software Engineer @ Mirantis
 
 
 On Sun, Aug 2, 2015 at 5:30 PM, Nikolay Makhotkin nmakhot...@mirantis.com 
 wrote:
 Hi guys! 
 
 That's awesome! It is very useful for all us!
 
 -- 
 Best Regards,
 Nikolay
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mistral] BPMN support

[Dmitri Zimine]

Hi Noy, 

The short answer is No, BPMN is currently not supported, and No, we didn’t hear 
requests to support it yet. It is architecturally feasible but is a substantial 
effort, and not on the current roadmap. 

The longer answer is: Both Mistral DSL and BPMN come from the common root of 
workflow body of knowledge, The Workflow Reference Model. However specifics 
choices of 1) supported workflow patterns 2) syntax to express and 3) visual 
representations and 4) target users are all different, given the differences in 
the domain. Business processes are quite complex, and business users are much 
less technical, this forced more complexity into the workflow and workflow 
definition language - to express more visually and abstract it from “the code”. 

Industry experience with applying workflow to IT automation shown that few 
patterns are commonly used in the field. And technical IT users don’t hesitate 
dealing with the code (rather hesitate dealing with graphical tools or any 
tools that constraint them). 

Mistral DSL tried to strike the balance between using of workflow as a powerful 
abstraction, and keeping it simple, with minimal patterns to support typical IT 
operations. 

Mistral as workflow service is architecturally capable of supporting another 
workflow language; it’s a matter of writing workflow handler that supports  
BMPN syntax and implements underlying workflow logic.

Practically, however, BPMN may be too far away: it’s XML not YAML, it’s a 
different way to refer data, therefore an implementation may reviel more places 
were adjustments will be required. 

Hope this helps.

Cheers, 
Dmitri.


There’s no plans 
On Jul 23, 2015, at 1:05 AM, ITZIKOWITZ, Noy (Noy) 
noy.itzikow...@alcatel-lucent.com wrote:

 Hi,
 We had a question from our OSS team about the level of support of BPMN in 
 Mistral,
 Is there any plan to include that support? Did you heard from other people on 
 the community around the need for that?
 
 Thanks,
 Noy
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [murano] [mistral] [yaql] Prepare to Yaql 1.0 release

[Dmitri Zimine]

This is great news Alex, was looking forward to it, will be happy to migrate 
Mistral. 

Some heads-up on what syntactically changed would be much appreciated to pass 
on to our users; 
we likely will catch much of them with Mistral tests, but some may bubble up. 

DZ. 

On Jul 27, 2015, at 2:04 AM, Alexander Tivelkov ativel...@mirantis.com wrote:

 Hi folks,
 
 We are finally ready to release the 1.0.0 version of YAQL. It is a
 huge milestone: the language finally looks the way we initially wanted
 it to look. The engine got completely rewritten, tons of new
 capabilities have been added. Here is a brief (and incomplete) list of
 new features and improvements:
 
 * Support for kwargs and keyword-only args (Py3)
 * Optional function arguments
 * Smart algorithm to find matching function overload without side effects
 * Ability to organize functions into layers
 * Configurable list of operators (left/right associative binary,
 prefix/suffix unary with precedence)
 * No global variables. There can be  more than one parser with
 different set of operators simultaneously
 * List literals ([a, b])
 * Dictionary literals ({ a = b})
 * Handling of escape characters in string literals
 * Verbatim strings (`...`) and double-quotes (...)
 * =~ and !~ operators in default configuration (similar to Perl)
 * - operator to pass context
 * Alternate operator names (for example '*equal' instead of '#operator_=')
  so that it will be possible to have different symbol for particular operator
  without breaking standard library that expects operator to have well
 known names
 * Set operations
 * Support for lists and dictionaries as a dictionary keys and set elements
 * New framework to decorate functions
 * Ability to distinguish between functions and methods
 * Switchable naming conventions
 * Unicode support
 * Execution options available to all invoked functions
 * Iterators limitation
 * Ability to limit memory consumption
 * Can work with custom context classes
 * It is possible to extend both parser and set of expression classes
 on user-side
 * It is possible to create user-defined types (also can be used for
 dependency injection)
 * Legacy yaql 0.2.x backward compatibility mode
 * Comprehensive standard library of functions
 * High unit test coverage
 * Delegate and lambda support, including higher order lambdas
 
 etc, etc.
 
 So, this is a big change.
 
 And as it always happens when moving from 0.something to 1.x the
 breaking changes are inevitable. We have included the backwards
 compatibility mode, but it may not address all the possible concerns.
 
 So.
 We have released a release candidate 1 of yaql 1.0.0 on pypi: [1]
 It includes all the new functionality and is likely to be identical to
 final release (that's why it is RC after all) and we strongly
 encourage all the yaql users (Murano and Mistral first of all) to try
 it and prepare migration patches to use it. When the final release
 is out, we'll update the global requirements to yaql = 1.0.0, which
 is likely to break all your gate checks unless you quickly land a
 migrating patch.
 
 Please email us any concerns or contact me (ativelkov) or Stan Lagun
 (slagun) directly in IRC (#murano) if you need some quick help on yaql
 1.0 or migrating from 0.2.x
 
 Happy yaqling!
 
 
 [1] https://pypi.python.org/pypi/yaql/1.0.0.0rc1
 
 
 --
 Regards,
 Alexander Tivelkov
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [murano] [mistral] [yaql] Prepare to Yaql 1.0 release

[Dmitri Zimine]

Thank you Stan! 

On Aug 4, 2015, at 3:07 PM, Stan Lagun sla...@mirantis.com wrote:

 Dmitry, this depends on how you're going to use yaql. yaql 1.0 has so called 
 legacy mode that is a layer on top of yaql that brings nearly full backward 
 compatibility including some of the things that were wrong in yaql 0.2. Use 
 this mode when backward compatibility is a must. Without it the following 
 changes may affect queries written for 0.2:
 
 1. There are no more tuples. foo(a = b) will mean now f(a='b') instead of 
 foo(tuple('a', 'b')) as it was in yaql 0.2. However dict(key1 = value1, key2 
 = value2)  and several other functions with the same syntax work exactly how 
 they used to work in 0.2
 2. Contradictory to the first point there are no more lists. Yaql works on 
 immutable structures (tuple, frozenset and yaql1 own FrozenDict). All 
 operations that seem to modify data in fact return modified copies. All input 
 data will be automatically converted to their immutable versions and 
 converted back upon completion. This has 2 side effects: a) if you have 
 custom yaql functions they should follow the same pattern b) even if your 
 input data were build from tuples etc output will still use lists (e.g. JSON 
 format)
 3. $dict.key will raise KeyError for missing keys instead of returning None. 
 Also key must be a valid keyword and cannot start with 2 underscores. There 
 are many other ways to retrieve dictionary value: $dict.get(value), 
 $dict.get(value, default), $dict[value], $dict[value, default]
 4. In yaql 0.2 every function could be called as a method. So $x.foo() was 
 the same as foo($x). In yaql 1.0 it is up to function to decide if it wants 
 to be a function (foo($x), method ($x.foo()) or extension method (both 
 syntax).Considering that yaql handles different function overloads 
 $x.foo() may be a valid expression and execute something completely different 
 from foo($x).
  Most of type-specific functions are declared as a methods. So you no longer 
 can say toUpper(str) but only str.toUpper()
 5. yaql 0.2 had very few functions. yaql 1.0 has huge (~260 
 functions/operators etc) standard library. However several of the 0.2 
 functions now have different signature or name, replaced with better 
 alternative or just accept additional (optional) parameters
 6. $collection[$  0] doesn't work anymore. Use $collection.select($  0) for 
 that
 
 
 
 Sincerely yours,
 Stan Lagun
 Principal Software Engineer @ Mirantis
 
 
 On Tue, Aug 4, 2015 at 9:57 PM, Dmitri Zimine dzim...@stackstorm.com wrote:
 This is great news Alex, was looking forward to it, will be happy to migrate 
 Mistral.
 
 Some heads-up on what syntactically changed would be much appreciated to pass 
 on to our users;
 we likely will catch much of them with Mistral tests, but some may bubble up.
 
 DZ.
 
 On Jul 27, 2015, at 2:04 AM, Alexander Tivelkov ativel...@mirantis.com 
 wrote:
 
  Hi folks,
 
  We are finally ready to release the 1.0.0 version of YAQL. It is a
  huge milestone: the language finally looks the way we initially wanted
  it to look. The engine got completely rewritten, tons of new
  capabilities have been added. Here is a brief (and incomplete) list of
  new features and improvements:
 
  * Support for kwargs and keyword-only args (Py3)
  * Optional function arguments
  * Smart algorithm to find matching function overload without side effects
  * Ability to organize functions into layers
  * Configurable list of operators (left/right associative binary,
  prefix/suffix unary with precedence)
  * No global variables. There can be  more than one parser with
  different set of operators simultaneously
  * List literals ([a, b])
  * Dictionary literals ({ a = b})
  * Handling of escape characters in string literals
  * Verbatim strings (`...`) and double-quotes (...)
  * =~ and !~ operators in default configuration (similar to Perl)
  * - operator to pass context
  * Alternate operator names (for example '*equal' instead of '#operator_=')
   so that it will be possible to have different symbol for particular 
  operator
   without breaking standard library that expects operator to have well
  known names
  * Set operations
  * Support for lists and dictionaries as a dictionary keys and set elements
  * New framework to decorate functions
  * Ability to distinguish between functions and methods
  * Switchable naming conventions
  * Unicode support
  * Execution options available to all invoked functions
  * Iterators limitation
  * Ability to limit memory consumption
  * Can work with custom context classes
  * It is possible to extend both parser and set of expression classes
  on user-side
  * It is possible to create user-defined types (also can be used for
  dependency injection)
  * Legacy yaql 0.2.x backward compatibility mode
  * Comprehensive standard library of functions
  * High unit test coverage
  * Delegate and lambda support, including higher order lambdas
 
  etc, etc.
 
  So, this is a big

Re: [openstack-dev] [Mistral][Murano] What's the latest/greatest on YAQL?

[Dmitri Zimine]

Thanks Stan, 

This release is few more months. How soon?  Are you planning to support 0.2 in 
the meantime?

We are really pressed on this transition to 1.0. 
For instance. Number 1 user error while dealing with YAQL is using == instead 
of =. 
We had this discussion and you and Alex and you suggested it’s easy to 
redefine. 

But… … The short is we need to move to 1.0 to do it. Because from what I 
figured so far, in 0.2 I can’t naively extend the an operator, as it won’t 
parse. I did make it work on 0.2 but this required a hack in the YAQL library 
itself, need to add ‘==‘ to both lexer.py and parser.py. At least what I 
figured. 

I see the tokens are already generalized in 1.0. It doesn’t seem to worth 
backporting it to 0.2, but if this is a way, let us know.

Mistral is betting on YAQL, please help.

DZ. 

On Jun 26, 2015, at 2:20 AM, Stan Lagun sla...@mirantis.com wrote:

 The plan is to move to yaql 1.0 this release.  Please do not merge yaql 1.0 
 into Mistral yet. Migration is going to happen really soon
 
 Sincerely yours,
 Stan Lagun
 Principal Software Engineer @ Mirantis
 
 
 On Fri, Jun 26, 2015 at 8:17 AM, Renat Akhmerov rakhme...@mirantis.com 
 wrote:
 Yes, it’s a pretty important thing that we’d like to clarify as soon as 
 possible.
 
 Any input from Murano team?
 
 Renat Akhmerov
 @ Mirantis Inc.
 
 
 
  On 26 Jun 2015, at 06:01, Dmitri Zimine dzim...@stackstorm.com wrote:
 
  Folks,
 
  it’s been some time,what’s the news:
 
  * Is Murano moving YAQL 1.0 in this cycle?
  * What’s your recommendation for this cycle - stay on 0.2.6 or move on?
  * Any progress on documentation?
 
  Thanks, DZ
 
 
  __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral][Murano] What's the latest/greatest on YAQL?

[Dmitri Zimine]

Folks, 

it’s been some time,what’s the news: 

* Is Murano moving YAQL 1.0 in this cycle?
* What’s your recommendation for this cycle - stay on 0.2.6 or move on? 
* Any progress on documentation? 

Thanks, DZ


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Proposal for the Resume Feature

[Dmitri Zimine]

+1 great write-up Winson,

I propose we move the discussion to an etherpad, and flash out details there so 
it won’t get lost in a long thread. 
Winson would you care to create one and post here? 

Re: ‘error state’: I think it’s not absolutely necessary: pause/resume can be 
done without enabling ‘error-running’ transition, 
by making default task policy `on-error: pause`so that if user chooses, the 
workflow goes into paused state on errors.
But it may be convenient, so no strong opinion on this yet. 


Re: checkpoint and roll-backs - yes! I see this and pause-resume complimentary. 
To be precise on terminology, workflows don't “roll-back” - this is more 
transactional term, they “compensate”, by running a ‘compensation workflow’ 
that gets system to back to a checkpoint state. 
At the end of compensational process the system goes in “paused” state where it 
can be resumed once the ‘cause of failure’ is fixed. 

DZ. 

On Jun 15, 2015, at 10:25 PM, BORTMAN, Limor (Limor) 
limor.bort...@alcatel-lucent.com wrote:

 +1,
 I just have one question. Do we want to able resume for WF  in error state?
 I mean isn't real resume it should be more of a rerun, don't you think?
 So in an error state we will create new executor and just re run it
 Thanks Limor
 
 
 
 -Original Message-
 From: Lingxian Kong [mailto:anlin.k...@gmail.com] 
 Sent: Tuesday, June 16, 2015 5:47 AM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Mistral] Proposal for the Resume Feature
 
 Thanks Winson for the write-up, very detailed infomation. (the format was 
 good)
 
 I'm totally in favor of your idea, actually, I really think you proposal is 
 complementary to my proposal in 
 https://etherpad.openstack.org/p/vancouver-2015-design-summit-mistral,
 please see 'Workflow rollback/recovery' section.
 
 What I wanna do is configure some 'checkpoints' throughout the workflow, and 
 if some task failed, we could rollback the execution to some checkpoint, and 
 resume the whole workflow after we have fixed some problem, seems like the 
 execution has never been failed before.
 
 It's just a initial idea, I'm waiting for our discussion to see if it really 
 makes sense to users, to get feedback, then we can talk about the 
 implementation and cooperation.
 
 On Tue, Jun 16, 2015 at 7:51 AM, W Chan m4d.co...@gmail.com wrote:
 Resending to see if this fixes the formatting for outlines below.
 
 
 I want to continue the discussion on the workflow resume feature.
 
 
 Resuming from our last conversation @
 http://lists.openstack.org/pipermail/openstack-dev/2015-March/060265.h
 tml. I don't think we should limit how users resume. There may be 
 different possible scenarios. User can fix the environment or 
 condition that led to the failure of the current task and the user 
 wants to just re-run the failed task.  Or user can actually fix the 
 environment/condition which include fixing what the task was doing, 
 then just want to continue the next set of task(s).
 
 
 The following is a list of proposed changes.
 
 
 1. A new CLI operation to resume WF (i.e. mistral workflow-resume).
 
A. If no additional info is provided, assume this WF is manually 
 paused and there are no task/action execution errors. The WF state is 
 updated to RUNNING. Update using the put method @ 
 ExecutionsController. The put method checks that there's no task/action 
 execution errors.
 
B. If WF is in an error state
 
i. To resume from failed task, the workflow-resume command 
 requires the WF execution ID, task name, and/or task input.
 
ii. To resume from failed with-items task
 
a. Re-run the entire task (re-run all items) requires WF
 execution ID, task name and/or task input.
 
b. Re-run a single item requires WF execution ID, task 
 name, with-items index, and/or task input for the item.
 
c. Re-run selected items requires WF execution ID, task 
 name, with-items indices, and/or task input for each items.
 
- To resume from the next task(s), the workflow-resume 
 command requires the WF execution ID, failed task name, output for the 
 failed task, and a flag to skip the failed task.
 
 
 2. Make ERROR - RUNNING as valid state transition @ 
 is_valid_transition function.
 
 
 3. Add a comments field to Execution model. Add a note that indicates 
 the execution is launched by workflow-resume. Auto-populated in this case.
 
 
 4. Resume from failed task.
 
A. Re-run task with the same task inputs  POST new action 
 execution for the task execution @ ActionExecutionsController
 
B. Re-run task with different task inputs  POST new action 
 execution for the task execution, allowed for different input @ 
 ActionExecutionsController
 
 
 5. Resume from next task(s).
 
A. Inject a noop task execution or noop action execution 
 (undecided yet) for the failed task with appropriate output.  The spec 
 is an adhoc spec that 

[openstack-dev] [Mistral] remind me about with_items concurrency

[Dmitri Zimine]

Folks, pls remind me where we ended up with ‘concurrency’? 

In the current implementation concurrency is a task policy (and not sure how 
much we tested it, not with unit testing/automated testing).

Also I recall discussing/ going back and forth re if concurrency is a task 
policy or it belongs to with_items, and at some point we admitted that Nikolay 
was right from the beginning when advocating for concurrency as with_item 
property. 

What’s the desired? 

This is not reopening the discussion: I just need to recall what the decision 
was :)

DZ. 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mistral] Break_on in Retry policy

[Dmitri Zimine]

may be not quite - please advice how it works in these cases 

1) if break-on expression contains the reference to task result, like 
break-on: % $.my_task.foo.bar = true % 
but action returns ERROR and task payload is None (desired behavior: don’t 
puke, evaluate to false and don’t break)

2) if break-on contains the value from (e.g. published variable, updated by 
other branch of workflow) - desired behavior - evaluate my_global_flag on every 
iteration: 
break-on %  $.my_global_flag = true %

3) a combination of the two
break-on %  $.my_global_counter  $.my_task.counter  %

We need functional tests for all 3 cases (may be unit tests but these cases 
become difficult to simulate/mock).

DZ. 

On Apr 22, 2015, at 6:55 AM, Nikolay Makhotkin nmakhot...@mirantis.com wrote:

 So, in this case I guess 'break-on' will work correctly now: 
 https://github.com/stackforge/mistral/blob/master/mistral/engine/policies.py#L295-L296
 
 On Wed, Apr 22, 2015 at 2:58 PM, Renat Akhmerov rakhme...@mirantis.com 
 wrote:
 Lingxian, yes, that’s basically what I suggest too.
 
 Renat Akhmerov
 @ Mirantis Inc.
 
 
 
  On 22 Apr 2015, at 16:03, Lingxian Kong anlin.k...@gmail.com wrote:
 
  Hi all,
 
  In my understanding, the meaning of the 'break-on' in 'retry' policy
  is just give an oppotunity to end the task execution earlier, i.e. we
  don't need to wait for all the iterations. I prefer that we keep the
  ERROR state, and keep it simple.
 
  On Wed, Apr 22, 2015 at 5:06 PM, Renat Akhmerov rakhme...@mirantis.com 
  wrote:
  Ok, after all thinking my suggestion is to leave break-on” but clarify its
  semantics and behavior a little bit as follow:
 
  As Dmitri wrote before “success-on” and “error-on” may be easily confused
  with “on-success” and “on-error”.
  “retry policy loop may only stop if:
 
  Task action/workflow completed successfully (no need to retry anymore). 
  This
  case has nothing to do with “break-on”.
  Task action/workflow failed and some condition (“break-on”) evaluates to
  True. So in this case I don’t think we need to give a user opportunity to
  change semantics of task behavior and be able to say “although task
  action/workflow failed I want this task to finish with success”. IMO, it 
  may
  be 1) confusing 2) error-prone 3) complicating our understanding of how
  workflow works. In other works, I’m now against of this behavior and I 
  think
  that success/error of action/workflow should exactly match to success/error
  of task.
 
  Based on the previous thoughts evaluation of “break-on” condition should
  work against task inbound context that doesn’t contain a task result itself
  (because the action failed) but may include results of other tasks 
  completed
  in parallel branches. The general use case for this is to “stop waiting for
  something if we see that fundamental requirements for that are not met”.
 
 
  Thoughts?
 
  Renat Akhmerov
  @ Mirantis Inc.
 
 
 
  On 21 Apr 2015, at 14:11, Nikolay Makhotkin nmakhot...@mirantis.com 
  wrote:
 
  Any more suggestions/thoughts here? Dmitri?
 
  More words: succeed-on / fail-on, success-expr / error-expr.
 
  --
  Best Regards,
  Nikolay
  __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
  __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
  --
  Regards!
  ---
  Lingxian Kong
 
  __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 -- 
 Best Regards,
 Nikolay
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] [Murano] Mistral devstack installation is failing in murano gate job

[Dmitri Zimine]

My 2c: 

Yes Mistral moved to YAQL 1.0 based on Murano team recommendations :)

some questions/comments before we decide how to proceed: 

1) Let’s clarify the impact: this problem doesn’t affect Murano directly; but 
it impacts Murano-Congress-Mistral initiative, correct? 
Is this a voting gate? What exactly is impacted? Are there any simpler 
workarounds? 

2) on YAQL readiness:
Mistral moved to YAQL it because 1) power 2) upcoming docs and 3) compatibility.

We target to claim Mistral DSL “complete” in Kilo. YAQL is a big part of DSL 
from the user standpoint.
Changing YAQL makes users migrate their workflows.
Thus we want to stick to a version of YAQL which will be documented and used 
long term. 

If YAQL 1.0 is not ready in Kilo we should revert no questions. 
If it is ready, and comes with documentation - would it be good for Murano 
users if Murano moves to it?

3) given that YAQL 0.2 is supported for another cycle (.5 year) and users of 
both Mistral and Murano are using it,
are there any plans to add documentation to it? It is the lack of docs on 0.2 
is the biggest reason to push forward. 
(Does this sound like an invitation to cheat and offer no docs for 1.0 in kilo 
to convince Mistral to stay on 0.2?)

DZ 

On Apr 13, 2015, at 6:13 AM, Serg Melikyan smelik...@mirantis.com wrote:

 Hi Nikolay  Filip,
 
 indeed, root cause of the issue is that Murano  Mistral use different
 version of yaql library. Murano installs yaql 0.2.4 and overrides
 1.0.0b2 already installed and expected by Mistral.
 
 We decided that we are not going to switch to the yaql 1.0.0 in Kilo
 since we already finished Kilo development and working on bug-fixes
 and releasing RC. This gate may be fixed if only Mistral will revert
 1.0.0 support in Kilo :'(
 
 Nikolay, what do you think about migrating to YAQL 1.0.0 in the next
 release? I know that it was me who proposed Mistral team to adopt yaql
 1.0.0, and I am sorry, I didn't realize all consequences of moving
 Mistral to yaql 1.0.0 and Murano team living with yaql 0.2.4.
 
 We need to work on packaging and supporting yaql in Ubuntu/CentOS in
 order to add this library to the global-requirements and to avoid this
 kind of issues in the future.
 
 On Mon, Apr 13, 2015 at 3:58 PM, Nikolay Makhotkin
 nmakhot...@mirantis.com wrote:
 
 We are facing an issue with Mistral devstack installation in our gate job 
 testing murano-congress-mistral integration (policy enforcement) [1] . 
 Mistral devstack scripts are failing with following import error [2]
 
 
 Hi, Filip!
 
 Recently Mistral has moved to new YAQL, and it seems this dependency is 
 missed (yaql 1.0, currently yaql 1.0.0b2)
 
 I think the root of problem is that Murano and Mistral have different yaql 
 versions installed.
 
 --
 Best Regards,
 Nikolay
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
 -- 
 Serg Melikyan, Senior Software Engineer at Mirantis, Inc.
 http://mirantis.com | smelik...@mirantis.com
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] Upgrading to YAQL 1.0

[Dmitri Zimine]

Renat, 

following up on the IRC meeting: I recalled one more item, and registered a 
blueprint - https://blueprints.launchpad.net/mistral/+spec/yaql-v1-0I think 
it’s good timing to do it in Kilo if YAQL 1.0 is announced; so I propose it for 
Kilo-rc1;

One question is when YAQL 1.0 is going to move to Pypi? we can get it from 
github link for some time but ideally should use pypi. Alex, Stan, what’s the 
plan? 

DZ.  


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] Propose Winson Chan m4dcoder for core team

[Dmitri Zimine]

Hi folks, 

I’d like to propose Winson Chan (m4dcoder) to become Mistral core team member. 

Winson brings unique mix of field experience implementing Mistral workflows in 
user environments, and solid development skills. 

He has been contributing to Mistral since Mar 2014. Winson did a 23 commits - a 
number of them are non-trivial, like moving RPC to oslo-messaging, or 
introducing environments, or making full validation. He has submitted 14 
blueprints and detailed design proposals, contributing to design and 
directions. He found, reported, and fixed bugs. He is becoming more active on 
the review board (would encourage to do more). 

I am +1 for m4dcoder as a core team member. 

DZ.





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack][Mistral] DSL improvements - certain and incubating

[Dmitri Zimine]

Team, thanks for your input so far, I flashed out one more section, please take 
a look and comment
https://docs.google.com/a/stackstorm.com/document/d/1Gy6V9YBt8W4llyErO_itHetkF1oNYv4ka-_5LdFKA18/edit#heading=h.n1jc8i9qhikt

DZ. 

On Mar 25, 2015, at 9:41 AM, Dmitri Zimine dzim...@stackstorm.com wrote:

 Folks, 
 
 we are discussing DSL improvements, based on Mistral field use and lessons 
 learned.
 
 Please join: comment on the document welcome, extra ideas, preferably based 
 on your experience writing Mistral workflows. 
 
 The summary is in the doc, it contains two sections:
 
 1) certain improvements, which we have complete clarity, agreed, and just 
 need to do
 2) incubating: ideas with more or less clear use case, where we however do 
 not have a certain, agreed solution.
 
 https://docs.google.com/a/stackstorm.com/document/d/1Gy6V9YBt8W4llyErO_itHetkF1oNYv4ka-_5LdFKA18/edit
 
 
 Regards, 
 Dmitri. 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TC][Mistral] Workflow Service project proposal

[Dmitri Zimine]

Hi respected TC members and community:

We would like to propose Workflow Service project (code name Mistral), 
as a project in the OpenStack namespace, in accordance with the new governance 
changes [1].

The details are on the review: https://review.openstack.org/#/c/170225

Please review the proposal at your earliest convenience, 
happy to answer questions and give more information here, or IRC on the 
#openstack-mistral.

Thanks, 

Dmitri Zimine (irc dzimine)

on behalf of Mistral contributors.

[1] http://governance.openstack.org/reference/new-projects-requirements.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Proposal for the Resume Feature

[Dmitri Zimine]

Thanks Winson for the summary. 

@Lingxian Kong
  The context for a task is used
 internally, I know the aim for this feature is to make it very easy
 and convinient for users to see the details for the workflow exection,
 but what users can do next with the context? Do you have a plan to
 change that context for a task by users? if the answer is no, I think
 it is not very necessary to expose the context endpoint.

I think the answer is “yes users will change the context” this falls out of use 
case #3. 
Let’s be specific: a create_vm task failed due to, say, network connection. 
As a user, I created the VM manually, now want to continue the workflow. 
Next step is attach storage to VM, needs VM ID published variable. So a user 
needs to 
modify outgoing context of create_vm task.

May be use case 2 be sufficient? 
We are also likely to specify multiple tasks: in case a parallel execution of 
two tasks
(create VM, create DNS record) failed again due to network conditions - than 
network 
is back I want to continue, but re-run those two exact tasks. 

Another point, may be obvious but let’s articulate it: we re-run task, not 
individual action within task.
In context of with_items, retry, repeat, it will lead to running actions 
multiple times.

Finally, workflow execution traceability. We need to get to the point of 
tracing pause and resume as workflow events. 

@Lingxian Kong
  we can introduce the notification
 system to Mistral, which is heavily used in other OpenStack projects.
care to elaborare? Thanks! 

DZ  


On Mar 26, 2015, at 10:29 PM, Lingxian Kong anlin.k...@gmail.com wrote:

 On Fri, Mar 27, 2015 at 11:20 AM, W Chan m4d.co...@gmail.com wrote:
 We assume WF is in paused/errored state when 1) user manually pause the WF,
 2) pause is specified on transition (on-condition(s) such as on-error), and
 3) task errored.
 
 The resume feature will support the following use cases.
 1) User resumes WF from manual pause.
 2) In the case of task failure, user fixed the problem manually outside of
 Mistral, and user wants to re-run the failed task.
 3) In the case of task failure, user fixed the problem manually outside of
 Mistral, and user wants to resume from the next task.
 this use case does really make sense to me.
 
 Resuming from #1 should be straightforward.
 Resuming from #2, user may want to change the inbound context.
 Resuming from #3, users is required to manually provide the published vars
 for the failed task(s).
 
 In our offline discussion, there's ambiguity with on-error clause and
 whether a task failure has already been addressed by the WF itself.  In many
 cases, the on-error tasks may just be logging, email notification, and/or
 other non-recovery procedures.  It's hard to determine that automatically,
 so we let users decide where to resume the WF instead.  Mistral will let
 user resume a WF from specific point. The resume function will determine the
 requirements needed to successfully resume.  If requirements are not met,
 then resume returns an error saying what requirements are missing.  In the
 case where there are failures in multiple parallel branches, the
 requirements may include more than one tasks.  For cases where user
 accidentally resume from an earlier task that is already successfully
 completed, the resume function should detect that and throw an exception.
 
 Also, the current change to separate task from action execution should be
 sufficient for traceability.
 
 We also want to expose an endpoint to let users view context for a task.
 This is to let user have a reference of the current task context to
 determine the delta they need to change for a successful resume.
 IMHO, I'm afraid I can't agree here. The context for a task is used
 internally, I know the aim for this feature is to make it very easy
 and convinient for users to see the details for the workflow exection,
 but what users can do next with the context? Do you have a plan to
 change that context for a task by users? if the answer is no, I think
 it is not very necessary to expose the context endpoint.
 
 However, considering the importance of context for the task
 execution(the resuming feature), we can introduce the notification
 system to Mistral, which is heavily used in other OpenStack projects.
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
 -- 
 Regards!
 ---
 Lingxian Kong
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [openstack][Mistral] DSL improvements - certain and incubating

[Dmitri Zimine]

Folks, 

we are discussing DSL improvements, based on Mistral field use and lessons 
learned.

Please join: comment on the document welcome, extra ideas, preferably based on 
your experience writing Mistral workflows. 

The summary is in the doc, it contains two sections:

1) certain improvements, which we have complete clarity, agreed, and just need 
to do
2) incubating: ideas with more or less clear use case, where we however do not 
have a certain, agreed solution.

https://docs.google.com/a/stackstorm.com/document/d/1Gy6V9YBt8W4llyErO_itHetkF1oNYv4ka-_5LdFKA18/edit


Regards, 
Dmitri. __
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] Mistral sublime plugin available

[Dmitri Zimine]

Mistral team got an exciting contribution: 

our good friend and industry veteran GP De Ciantis had built a sublime plugin 
for Mistral DSL - workbooks, workflows, etc. 

Check it out, and enjoy writing Mistral workflows! 

https://github.com/giampierod/mistral-sublime

Thanks GP for an excellent contrib! 

Regards, 
DZ.__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Changing expression delimiters in Mistral DSL

[Dmitri Zimine]

Zane, Angus, thanks for your input!

This functional based syntax is consisted for dev use. The trouble is it ITSELF 
needs to be delimited. Without pissing off the users :) Consider two key usage: 
 input parameters and shorthand syntax for action input. That’s why we are 
looking for two-char symmetric (opening + closing) delimiters. 

task_with_full_syntax_input:
  action: std.ssh
  input: 
 cmd = awk '{ print % $.my_var % }' /etc/passwd

task_with_shorthand_action_input:
  action: std.ssh cmd=awk '{ print \”% $.my_var %\ }' /etc/passwd”

using function-like we still will have to do 
  action: std.ssh cmd=“awk '{ print \”% yaql {$.my_var} %\ }' /etc/passwd”
that only adds confusion IMO. 

The full set of usages for YAQL in DSL is here: 
https://etherpad.openstack.org/p/mistral-YAQL-delimiters

DZ. 

On Feb 18, 2015, at 7:22 AM, Zane Bitter zbit...@redhat.com wrote:

 On 16/02/15 16:06, Dmitri Zimine wrote:
 2) Use functions, like Heat HOT or TOSCA:
 
 HOT templates and TOSCA doesn’t seem to have a concept of typed
 variables to borrow from (please correct me if I missed it). But they
 have functions: function: { function_name: {foo: [parameter1, parameter
 2], bar:xxx”}}. Applied to Mistral, it would look like:
 
 publish:
  - bool_var: { yaql: “1+1+$.my.var  100” }
 
 Not bad, but currently rejected as it reads worse than delimiter-based
 syntax, especially in simplified one-line action invocation.
 
 Note that you don't actually need the quotes there, so this would be 
 equivalent:
 
publish:
 - bool_var: {yaql: 1+1+$.my.var  100}
 
 FWIW I am partial to this or to Renat's p7 suggestion:
 
publish:
 - bool_var: yaql{1+1+$.my.var  100}
 
 Both offer the flexibility to introduce new syntax in the future without 
 breaking backwards compatibility.
 
 cheers,
 Zane.
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Changing expression delimiters in Mistral DSL

[Dmitri Zimine]

}   # Here it’s bad that $ sign is used for two 
 different things
 p2: ~{1 + $.var}   # ~ is easy to miss in a text
 p3: ^{1 + $.var}   # For someone may be associated with regular 
 expressions
 p4: ?{1 + $.var}   
 p5: {1 + $.var}  # This is kinda crazy
 p6: e{1 + $.var}   # That looks a pretty interesting option to me, “e” 
 could mean “expression” here.
 p7: yaql{1 + $.var}# This is interesting because it would give a 
 clear and easy mechanism to plug in other expression languages, “yaql” 
 here is a used dialect for the following expression
 p8: y{1 + $.var}   # “y” here is just shortened “yaql
 
 
 Any ideas and thoughts would be really appreciated!
 
 Renat Akhmerov
 @ Mirantis Inc.
 
 
 
 On 17 Feb 2015, at 12:53, Renat Akhmerov rakhme...@mirantis.com wrote:
 
 Dmitri,
 
 I agree with all your reasonings and fully support the idea of changing 
 the syntax now as well as changing system’s API a little bit due to 
 recently found issues in the current engine design that don’t allow us, 
 for example, to fully implement ‘with-items’ (although that’s a little 
 bit different story).
 
 Just a general note about all changes happening now: Once we release kilo 
 stable release our API, DSL of version 2 must be 100% stable. I was 
 hoping to stabilize it much earlier but the start of production use 
 revealed a number of things (I think this is normal) which we need to 
 address, but not later than the end of Kilo.
 
 As far as % % syntax. I see that it would solve a number of problems 
 (YAML friendliness, type ambiguity) but my only not strong argument is 
 that it doesn’t look that elegant in YAML as it looks, for example, in 
 ERB templates. It really reminds me XML/HTML and looks like a bear in a 
 grocery store (tried to make it close to one old russian saying :) ). So 
 just for this only reason I’d suggest we think about other alternatives, 
 maybe not so familiar to Ruby/Chef/Puppet users but looking better with 
 YAML and at the same time being YAML friendly.
 
 I would be good if we could here more feedback on this, especially from 
 people who started using Mistral.
 
 Thanks
 
 Renat Akhmerov
 @ Mirantis Inc.
 
 
 
 On 17 Feb 2015, at 03:06, Dmitri Zimine dzim...@stackstorm.com wrote:
 
 SUMMARY: 
 
 
 We are changing the syntax for inlining YAQL expressions in Mistral YAML 
 from {1+$.my.var} (or “{1+$.my.var}”) to % 1+$.my.var %
 
 Below I explain the rationale and the criteria for the choice. Comments 
 and suggestions welcome.
 
 DETAILS: 
 -
 
 We faced a number of problems with using YAQL expressions in Mistral 
 DSL: [1] must handle any YAQL, not only the ones started with $; [2] 
 must preserve types and [3] must comply with YAML. We fixed these 
 problems by applying Ansible style syntax, requiring quotes around 
 delimiters (e.g. “{1+$.my.yaql.var}”). However, it lead to unbearable 
 confusion in DSL readability, in regards to types:
 
 publish:
intvalue1: {1+1}” # Confusing: you expect quotes to be string.
intvalue2: {int(1+1)}” # Even this doestn’ clean the confusion
whatisthis:{$.x + $.y}” # What type would this return? 
 
 We got a very strong push back from users in the filed on this syntax. 
 
 The crux of the problem is using { } as delimiters YAML. It is plain 
 wrong to use the reserved character. The clean solution is to find a 
 delimiter that won’t conflict with YAML.
 
 Criteria for selecting best alternative are: 
 1) Consistently applies to to all cases of using YAML in DSL
 2) Complies with YAML 
 3) Familiar to target user audience - openstack and devops
 
 We prefer using two-char delimiters to avoid requiring extra escaping 
 within the expressions.
 
 The current winner is % %. It fits YAML well. It is familiar to 
 openstack/devops as this is used for embedding Ruby expressions in 
 Puppet and Chef (for instance, [4]). It plays relatively well across all 
 cases of using expressions in Mistral (see examples in [5]):
 
 ALTERNATIVES considered:
 --
 
 1) Use Ansible-like syntax: 
 http://docs.ansible.com/YAMLSyntax.html#gotchas
 Rejected for confusion around types. See above.
 
 2) Use functions, like Heat HOT or TOSCA:
 
 HOT templates and TOSCA doesn’t seem to have a concept of typed 
 variables to borrow from (please correct me if I missed it). But they 
 have functions: function: { function_name: {foo: [parameter1, parameter 
 2], bar:xxx”}}. Applied to Mistral, it would look like:
 
 publish:
  - bool_var: { yaql: “1+1+$.my.var  100” } 
 
 Not bad, but currently rejected as it reads worse than delimiter-based 
 syntax, especially in simplified one-line action invocation.
 
 3)   paired with other symbols: php-styoe  ? ..?
 
 
 REFERENCES: 
 --
 
 [1] Allow arbitrary YAQL expressions, not just ones started with $ : 
 https://github.com/stackforge/mistral/commit/5c10fb4b773cd60d81ed93aec33345c0bf8f58fd
 [2] Use

[openstack-dev] [Mistral] Changing expression delimiters in Mistral DSL

[Dmitri Zimine]

SUMMARY: 


We are changing the syntax for inlining YAQL expressions in Mistral YAML from 
{1+$.my.var} (or “{1+$.my.var}”) to % 1+$.my.var %

Below I explain the rationale and the criteria for the choice. Comments and 
suggestions welcome.

DETAILS: 
-

We faced a number of problems with using YAQL expressions in Mistral DSL: [1] 
must handle any YAQL, not only the ones started with $; [2] must preserve types 
and [3] must comply with YAML. We fixed these problems by applying Ansible 
style syntax, requiring quotes around delimiters (e.g. “{1+$.my.yaql.var}”). 
However, it lead to unbearable confusion in DSL readability, in regards to 
types:

publish:
   intvalue1: {1+1}” # Confusing: you expect quotes to be string.
   intvalue2: {int(1+1)}” # Even this doestn’ clean the confusion
   whatisthis:{$.x + $.y}” # What type would this return? 

We got a very strong push back from users in the filed on this syntax. 

The crux of the problem is using { } as delimiters YAML. It is plain wrong to 
use the reserved character. The clean solution is to find a delimiter that 
won’t conflict with YAML.

Criteria for selecting best alternative are: 
1) Consistently applies to to all cases of using YAML in DSL
2) Complies with YAML 
3) Familiar to target user audience - openstack and devops

We prefer using two-char delimiters to avoid requiring extra escaping within 
the expressions.

The current winner is % %. It fits YAML well. It is familiar to 
openstack/devops as this is used for embedding Ruby expressions in Puppet and 
Chef (for instance, [4]). It plays relatively well across all cases of using 
expressions in Mistral (see examples in [5]):

ALTERNATIVES considered:
--

1) Use Ansible-like syntax: http://docs.ansible.com/YAMLSyntax.html#gotchas
Rejected for confusion around types. See above.

2) Use functions, like Heat HOT or TOSCA:

HOT templates and TOSCA doesn’t seem to have a concept of typed variables to 
borrow from (please correct me if I missed it). But they have functions: 
function: { function_name: {foo: [parameter1, parameter 2], bar:xxx”}}. 
Applied to Mistral, it would look like:

publish:
 - bool_var: { yaql: “1+1+$.my.var  100” } 

Not bad, but currently rejected as it reads worse than delimiter-based syntax, 
especially in simplified one-line action invocation.

3)   paired with other symbols: php-styoe  ? ..?


REFERENCES: 
--

[1] Allow arbitrary YAQL expressions, not just ones started with $ : 
https://github.com/stackforge/mistral/commit/5c10fb4b773cd60d81ed93aec33345c0bf8f58fd
[2] Use Ansible-like syntax to make YAQL expressions YAML complient 
https://github.com/stackforge/mistral/commit/d9517333b1fc9697d4847df33d3b774f881a111b
[3] Preserving types in YAQL 
https://github.com/stackforge/mistral/blob/d9517333b1fc9697d4847df33d3b774f881a111b/mistral/tests/unit/test_expressions.py#L152-L184
[4]Using % % in Puppet 
https://docs.puppetlabs.com/guides/templating.html#erb-is-plain-text-with-embedded-ruby
 
[5] Etherpad with discussion 
https://etherpad.openstack.org/p/mistral-YAQL-delimiters
[6] Blueprint https://blueprints.launchpad.net/mistral/+spec/yaql-delimiters

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Murano][Heat][Mistral] Use and adoption of YAQL

[Dmitri Zimine]

Stan, Alex, Renat: 

Should we migrate to YAQL 1.0 now? and stop using the initial one? What’s the 
delta? 

Still short on the docs :) but I understand they’re coming up. 
https://github.com/stackforge/yaql/tree/master/doc/source

Cheers, Dmitri. 

On Jan 16, 2015, at 6:46 AM, Stan Lagun sla...@mirantis.com wrote:

 Dmitri,
 
 we are working hard towards stable YAQL 1.0 which is expected to be released 
 during Kilo cycle. It is going to have proper documentation and high unit 
 test coverage which can also serve as a documentation source. YAQL has 
 already migrated to StackForge and adopted OpenStack development process and 
 tools but the work is still in progress. Any help from Mistral team and/or 
 other YAQL adopters is appreciated.
 
 
 
 Sincerely yours,
 Stan Lagun
 Principal Software Engineer @ Mirantis
 
 
 On Thu, Jan 15, 2015 at 10:54 PM, Dmitri Zimine dzim...@stackstorm.com 
 wrote:
 Folks, 
 
 We use YAQL in Mistral for referencing variables, expressing conditions, etc. 
 Murano is using it extensively, I saw Heat folks thought of using it, at 
 least once :) May be others...
 
 We are learning that YAQL incredibly powerful comparing to alternatives like 
 Jinja2 templates used  in salt / ansible. 
 
 But with lack of documentation, it becomes one of adoption blockers to 
 Mistral (we got very vocal user feedback on this).
 
 This is pretty much all the docs I can offer our users on YAQL so far. Not 
 much. 
 http://yaql.readthedocs.org/en/latest/
 https://github.com/ativelkov/yaql/blob/master/README.rst
 https://murano.readthedocs.org/en/latest/murano_pl/murano_pl.html#yaql
 
 Are there any plans to fix it? 
 
 Are there interest from other projects to use YAQL? 
 
 Cheers, 
 DZ. 
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Murano][Heat][Mistral] Use and adoption of YAQL

[Dmitri Zimine]

Folks, 

We use YAQL in Mistral for referencing variables, expressing conditions, etc. 
Murano is using it extensively, I saw Heat folks thought of using it, at least 
once :) May be others...

We are learning that YAQL incredibly powerful comparing to alternatives like 
Jinja2 templates used  in salt / ansible. 

But with lack of documentation, it becomes one of adoption blockers to Mistral 
(we got very vocal user feedback on this).

This is pretty much all the docs I can offer our users on YAQL so far. Not 
much. 
http://yaql.readthedocs.org/en/latest/
https://github.com/ativelkov/yaql/blob/master/README.rst
https://murano.readthedocs.org/en/latest/murano_pl/murano_pl.html#yaql

Are there any plans to fix it? 

Are there interest from other projects to use YAQL? 

Cheers, 
DZ. __
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] Access to worklfow/task results without implicit publish

[Dmitri Zimine]

The problem:

Refer to workflow / action output without explicitly re-publishing the output 
values. Why we want it: to reduce repetition, and to make modifications in the 
place values are used, not where they are obtained (and not in multiple 
places). E.g., as an editor of a workflow, when I just realized that I need a 
value of some task down the line, I want to make change right here in the tasks 
that consumes the data (and only those which need this data), without finding 
and modifying the task that supplies the data.

Reasons:

We don't have a concept of workflow or action ‘results': it's the task which 
produces and publishes results. Different tasks call same actions/workflows, 
produce same output variables with diff values. We don't want to publish this 
output with output name as a key, to the global context: they will conflict and 
mess up. Instead, we can namespace them by the task (as specific values are the 
attributes of the tasks, and we want to refer to tasks, not actions/workflows).

Solution:

To refer the output of a particular task (aka raw result of action execution 
invoked by this task), use the_task prefix:

 $_task.taskname.path.to.variable
 $_task.my_task.my_task_result.foo.bar


Expanded example
 
my_sublfow:
   output:
- foo #  declare output here
- bar 
   tasks:
   my_task:
 action: get_foo
 publish: 
 foo: $foo #  define output in a task
 bar: $bar
 ...
main_flow_with_explicit_publishing:
tasks:
t1: 
   workflow: my_subflow 
publish: 
   # Today, you must explicitly publish to make data 
   # from action available for other tasks
foo: $foo #  re-plublish, else you can't use it
bar: $bar
t2: 
action: echo output=$foo and $bar #  use it from task t1

main_flow_with_implicit_publishing:
tasks:
t1: 
   workflow: my_subflow 
t2: 
action: echo output=$_task.t1.foo and $_task.t1.bar

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] For-each

[Dmitri Zimine]

Thanks Angus. 

One obvious thing is we either make it somewhat consistent, or name it 
differently. 
These looks similar, at least on the surface. I wonder if the feedback we’ve 
got so far (for-each is confusing because it brings wrong expectations) is 
applicable to Heat, too. 

Another observation on naming consistency - mistral uses dash, like `for-each`. 
Heat uses _underscores when naming YAML keys. 
So does TOSCA standard. We should have thought about this earlier but it may be 
not late to fix it while v2 spec is still forming. 

DZ. 


On Dec 18, 2014, at 9:07 PM, Angus Salkeld asalk...@mirantis.com wrote:

 On Mon, Dec 15, 2014 at 8:00 PM, Nikolay Makhotkin nmakhot...@mirantis.com 
 wrote:
 Hi,
 
 Here is the doc with suggestions on specification for for-each feature.
 
 You are free to comment and ask questions.
 
 https://docs.google.com/document/d/1iw0OgQcU0LV_i3Lnbax9NqAJ397zSYA3PMvl6F_uqm0/edit?usp=sharing
 
 
 
 Just as a drive by comment, there is a Heat spec for a for-each: 
 https://review.openstack.org/#/c/140849/
 (there hasn't been a lot of feedback for it yet tho')
 
 Nice to have these somewhat consistent.
 
 -Angus
  
 
 -- 
 Best Regards,Nikolay
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Plans to load and performance testing

[Dmitri Zimine]

Anastasia, any start is a good start. 

 1 api 1 engine 1 executor, list-workbooks

what exactly doest it mean: 1) is mistral deployed on 3 boxes with component 
per box, or all three are processes on the same box? 2) is list-workbooks test 
running while workflow executions going on? How many? what’s the character of 
the load 3) when it says 60% success what exactly does it mean, what kind of 
failures? 4) what is the durability criteria, how long do we expect Mistral to 
withstand the load.  

Let’s discuss this in details on the next IRC meeting? 

Thanks again for getting this started. 

DZ.


On Dec 19, 2014, at 7:44 AM, Anastasia Kuznetsova akuznets...@mirantis.com 
wrote:

 Boris,
 
 Thanks for feedback! 
 
  But I belive that you should put bigger load here: 
  https://etherpad.openstack.org/p/mistral-rally-testing-results
 
 As I said it is only beginning and  I will increase the load and change its 
 type. 
 
 As well concurrency should be at least 2-3 times bigger than times otherwise 
 it won't generate proper load and you won't collect enough data for 
 statistical analyze.  
 
 As well use  rps runner that generates more real life load.
 Plus it will be nice to share as well output of rally task report command.
 
 Thanks for the advice, I will consider it in further testing and reporting.
 
 Answering to your question about using Rally for integration testing, as I 
 mentioned in our load testing plan published on wiki page,  one of our final 
 goals is to have a Rally gate in one of Mistral repositories, so we are 
 interested in it and I already prepare first commits to Rally.
 
 Thanks,
 Anastasia Kuznetsova
 
 On Fri, Dec 19, 2014 at 4:51 PM, Boris Pavlovic bpavlo...@mirantis.com 
 wrote:
 Anastasia, 
 
 Nice work on this. But I belive that you should put bigger load here: 
 https://etherpad.openstack.org/p/mistral-rally-testing-results
 
 As well concurrency should be at least 2-3 times bigger than times otherwise 
 it won't generate proper load and you won't collect enough data for 
 statistical analyze.  
 
 As well use  rps runner that generates more real life load.
 Plus it will be nice to share as well output of rally task report command.
 
 
 By the way what do you think about using Rally scenarios (that you already 
 wrote) for integration testing as well? 
 
 
 Best regards,
 Boris Pavlovic 
 
 On Fri, Dec 19, 2014 at 2:39 PM, Anastasia Kuznetsova 
 akuznets...@mirantis.com wrote:
 Hello everyone,
 
 I want to announce that Mistral team has started work on load and performance 
 testing in this release cycle.
 
 Brief information about scope of our work can be found here: 
 https://wiki.openstack.org/wiki/Mistral/Testing#Load_and_Performance_Testing
 
 First results are published here:
 https://etherpad.openstack.org/p/mistral-rally-testing-results
 
 Thanks,
 Anastasia Kuznetsova
 @ Mirantis Inc.
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] For-each

[Dmitri Zimine]

Based on the feedback so far, I updated the document and added some more 
details from the comments and discussions. 

We still think for-each as a keyword confuses people by setting up some 
behavior expectations (e.g., it will run sequentially, you can work with data 
inside the loop, you can ‘nest’ for-each loops - while it’s not a loop at all, 
just a way to run actions which are not accepting arrays of data, with arrays 
of data). 

But no better idea on the keyword just yet. 

DZ. 


On Dec 15, 2014, at 10:53 PM, Renat Akhmerov rakhme...@mirantis.com wrote:

 Thanks Nikolay,
 
 I also left my comments and tend to like Alt2 better than others. Agree with 
 Dmitri that “all-permutations” thing can be just a different construct in the 
 language and “concurrency” should be rather a policy than a property of 
 “for-each” because it doesn’t have any impact on workflow logic itself, it 
 only influence the way how engine runs a task. So again, policies are engine 
 capabilities, not workflow ones.
 
 One tricky question that’s still in the air is how to deal with publishing. I 
 mean in terms of requirements it’s pretty clear: we need to apply “publish” 
 once after all iterations and be able to access an array of iteration results 
 as $. But technically, it may be a problem to implement such behavior, need 
 to think about it more.
 
 Renat Akhmerov
 @ Mirantis Inc.
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] RFC - Action spec CLI

[Dmitri Zimine]

The problem with existing syntax is it is not defined: there is no docs on 
inlining complex variables [*], and we haven’t tested it for anything more than 
the simplest cases: 
https://github.com/stackforge/mistral/blob/master/mistral/tests/unit/workbook/v2/test_dsl_specs_v2.py#L114.
 I will be surprised if anyone figured how to provide a complex object as an 
inline parameter.

Do you think regex is the right approach for parsing arbitrary key-values where 
values is arbitrary json structures? Will it work with something like 
workflow: wf2 object_list=[ {“url”: 
“http://{$hostname}.example.com:8080?x=ay={$.b}}, 33, null, {{$.key}, 
[{$.value1}, {$.value2}]}
How much tests should we write to be confident we covered all cases? I share 
Lakshmi’s concern it is fragile and maintaining it reliably is difficult. 

But back to the original question, it’s about requirements, not implementation. 
My preference is “option 3”, “make it work as is now”. But if it’s too hard I 
am ok to compromise. 
Than option 2 as it resembles option 3 and YAML/JSON conversion makes complete 
sense. At the expense of quoting the objects. Slight change, not significant. 
Option 1 introduce a new syntax; although familiar to CLI users, I think it’s a 
bit out of place in YAML definition. 
Option 4 is no go :)

DZ. 

[*] “there is no docs to this” - I subscribe on fixing this.

On Dec 16, 2014, at 9:48 PM, Renat Akhmerov rakhme...@mirantis.com wrote:

 Ok, I would prefer to spend some time and think how to improve the existing 
 reg exp that we use to parse key-value pairs. We definitely can’t just drop 
 support of this syntax and can’t even change it significantly since people 
 already use it.
 
 Renat Akhmerov
 @ Mirantis Inc.
 
 
 
 On 17 Dec 2014, at 07:28, Lakshmi Kannan laks...@lakshmikannan.me wrote:
 
 Apologies for the long email. If this fancy email doesn’t render correctly 
 for you, please read it here: 
 https://gist.github.com/lakshmi-kannan/cf953f66a397b153254a
 
 I was looking into fixing bug: 
 https://bugs.launchpad.net/mistral/+bug/1401039. My idea was to use shlex to 
 parse the string. This actually would work for anything that is supplied in 
 the linux shell syntax. Problem is this craps out when we want to support 
 complex data structures such as arrays and dicts as arguments. I did not 
 think we supported a syntax to take in complex data structures in a one line 
 format. Consider for example:
 
   task7:
 for-each:
   vm_info: $.vms
 workflow: wf2 is_true=true object_list=[1, null, str]
 on-complete:
   - task9
   - task10
 Specifically
 
 wf2 is_true=true object_list=[1, null, str]
 shlex will not handle this correctly because object_list is an array. Same 
 problem with dict.
 
 There are 3 potential options here:
 
 Option 1
 
 1) Provide a spec for specifying lists and dicts like so:
 list_arg=1,2,3 and dict_arg=h1:h2,h3:h4,h5:h6
 
 shlex will handle this fine but there needs to be a code that converts the 
 argument values to appropriate data types based on schema. (ActionSpec 
 should have a parameter schema probably in jsonschema). This is doable.
 
 wf2 is_true=true object_list=1,null,str
 Option 2
 
 2) Allow JSON strings to be used as arguments so we can json.loads them (if 
 it fails, use them as simple string). For example, with this approach, the 
 line becomes
 
 wf2 is_true=true object_list=[1, null, str]
 This would pretty much resemble 
 http://stackoverflow.com/questions/7625786/type-dict-in-argparse-add-argument
 
 Option 3
 
 3) Keep the spec as such and try to parse it. I have no idea how we can do 
 this reliably. We need a more rigorous lexer. This syntax doesn’t translate 
 well when we want to build a CLI. Linux shells cannot support this syntax 
 natively. This means people would have to use shlex syntax and a translation 
 needs to happen in CLI layer. This will lead to inconsistency. CLI uses some 
 syntax and the action input line in workflow definition will use another. We 
 should try and avoid this.
 
 Option 4
 
 4) Completely drop support for this fancy one line syntax in workflow. This 
 is probably the least desired option.
 
 My preference
 
 Looking the options, I like option2/option 1/option 4/option 3 in the order 
 of preference.
 
 With some documentation, we can tell people why this is hard. People will 
 also grok because they are already familiar with CLI limitations in linux.
 
 Thoughts?
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org

Re: [openstack-dev] [Mistral] For-each

[Dmitri Zimine]

I had a short user feedback sessions with Patrick and James, the short summary 
is:

1) simplify the syntax to optimize for the most common use case
2) 'concurrency' is the best word - but bring it out of for-each to task level, 
or task/policies 
3) all-permutation - relatively rare case, either implement as a different 
construct - like ‘for-every’, or use workaround.

Another feedback is  for-each as a term is confusing: people expect a different 
behavior (e.g., run sequentially, modify individual elements)
while it is effectively a ‘map’ function. No good suggestion on the better name 
yet. Keep on looking.

The details are added into the document.

DZ. 

On Dec 15, 2014, at 2:00 AM, Nikolay Makhotkin nmakhot...@mirantis.com wrote:

 Hi,
 
 Here is the doc with suggestions on specification for for-each feature.
 
 You are free to comment and ask questions.
 
 https://docs.google.com/document/d/1iw0OgQcU0LV_i3Lnbax9NqAJ397zSYA3PMvl6F_uqm0/edit?usp=sharing
 
 
 
 -- 
 Best Regards,
 Nikolay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Global Context and Execution Environment

[Dmitri Zimine]

Winson, Lakshmi, Renat: 

It looked good and I began to write down the summary: 
https://etherpad.openstack.org/p/mistral-global-context

But than realized that it’s not safe to assume from the action, that the global 
context will be supplied as part of API call. 
Check it out in the etherpad.

What problems are we trying to solve: 
1) reduce passing the same parameters over and over from parent to child
2) “automatically” make a parameter accessible to most actions without typing 
it all over (like auth token) 

Can #1 be solved by passing input to subworkflows automatically
Can #2 be solved somehow else? Default passing of arbitrary parameters to 
action seems like breaking abstraction.

Thoughts? need to brainstorm further….

DZ 


On Dec 12, 2014, at 12:54 PM, W Chan m4d.co...@gmail.com wrote:

 Renat, Dmitri,
 
 On supplying the global context into the workflow execution...
 
 In addition to Renat's proposal, I have a few here.
 
 1) Pass them implicitly in start_workflow as another kwargs in the **params.  
 But on thinking, we should probably make global context explicitly defined in 
 the WF spec.  Passing them implicitly may be hard to follow during 
 troubleshooting where the value comes from by looking at the WF spec.  Plus 
 there will be cases where WF authors want it explicitly defined. Still 
 debating here...
 
 inputs = {...}
 globals = {...}
 start_workflow('my_workflow', inputs, globals=globals)
 
 2) Instead of adding to the WF spec, what if we change the scope in existing 
 input params?  For example, inputs defined in the top workflow by default is 
 visible to all subflows (pass down to workflow task on run_workflow) and 
 tasks (passed to action on execution).
 
 3) Add to the WF spec
 
 workflow:
 type: direct
 global:
 - global1
 - global2
 input:
 - input1
 - input2
 
 Winson
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Mistral - Real Time integration

[Dmitri Zimine]

Hi Raanan, 

1) yes, the workflow execution can be triggered by the REST API call. 

2) yes the Mistral is the closest in OpenStack for the higher-level automation 
and integration across operations and with external systems.

Are you coming to OpenStack summit by any chance? Your comments expose your 
experience in the area, we would love to meet and chat more. Else, let’s follow 
up at length here on the mail list.

Regards, DZ. 


On Oct 30, 2014, at 4:22 PM, Raanan Jossefi (RJ) Benikar rbeni...@gmail.com 
wrote:

 
 
 Hi,
 
 I see that Mistral can perform workflow automation and task execution per 
 scheduling but wanted to confirm if workflows can be executed in real-time 
 on-demand via REST API?   Can a REST API call into Mistral execute the 
 workflow upon request, is what I am confirming, as it seems it is the case. 
 Besides on-demand, is the purpose of Triggers currently being developed to 
 automatically detect changes or notifications and execute in response to 
 these triggers?  If so I would highly recommend triggers based on database 
 change log, on file changes , on message arrival inside an AMQP queue, and 
 on polling a REST API in which you expect a certain result, or status 
 message.
 
 Secondly, if one has a very tailored and mature cloud orchestration and 
 operations process in place and wants to migrate to OpenStack and offer 
 similar automation integrating with external change management systems, 
 performing secondary LDAP searches, performing multiple SOR / DB queries, 
 and thus interacting with other external non-cloud related technologies as 
 part of the process of creating new tenants, adding users, creating new 
 images, and creating and deploying machine instances would mistral be the 
 best mechanism for this, in terms of driving automation, and invoking 3rd 
 party APIs during and as part of the process?   
 
 My current use case looks like this:
 
 1) Tenant requests VM with specific properties (image, etc etc) -- Service 
 Now
 2) Service Now --- API ( based on properties/inputs ) query a DB to 
 automatically generate a server name )
 3) API --- OpenStack API to provision new VM.
 
 
 This is just an example, during this exchange, the API will interact with 
 several external systems besides a DB, and do more than just autogenerate 
 custom and unique machine names. In this case, the API could be Mistral that 
 I am calling via REST.  I would create API wrappers to the other components 
 ( which for the most part, I already have) and Mistral would now call both 
 wrapper APIs and OpenStack APIs to provision a machine via Nova with all of 
 the dependencies met.
 
 Your advice is kindly appreciated,
 Raanan 
 
 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] Moving docs from wiki to mistral/doc

[Dmitri Zimine]

Folks, I have moved the DSL and API docs to the project (up on review). Now it 
contains enough examples to get going, also see doc/README.md for some useful 
hints. 

Renat, Nikolay, if you find any omissions please add; and once we are happy 
let’s remove the DSL and API wiki to avoid confusion. Wiki can refer to the 
readthedocs page (generated with every build).

DZ ___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mistral] Mistral 0.2 planning

[Dmitri Zimine]

Thanks Renat for running and capturing.

One addition - allocate time for bug fixes, we’ll have quite a few :)

DZ.

On Oct 2, 2014, at 9:56 PM, Renat Akhmerov rakhme...@mirantis.com wrote:

 Hi,
 
 Mistral team has selected blueprints for Mistral 0.2 which is currently 
 scheduled for 10/31/2014 (may slightly change). Below are the links to 
 release LP page and etherpad with our estimations
 https://launchpad.net/mistral/+milestone/0.2
 https://etherpad.openstack.org/p/mistral-0.2-planning
 
 Please join the discussion if you have something to add/comment or if you’d 
 like to contribute to Mistral 0.2.
 
 Thanks
 
 Renat Akhmerov
 @ Mirantis Inc.
 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] callback on workflow completion

[Dmitri Zimine]

Use case: 
The client software fires the workflow execution and needs to be know when the 
workflow is complete. There is no good pool strategy as workflow can take 
arbitrary time from ms to days. Callback notification is needed. 

Solution is a webhook

Option 1: pass callback URL as part of starting workflow execution:
POST /executions
workflow_name=flow
callback= {
events: [[on-task-complete, on-execution-complete]
url: http://bla.com
method:POST
headers: {}
… other stuff to form proper HTTP call, like API tokens, etc ...
}
…..


Option 2: webhook endpoint
Mistral exposes /webhook controller 
Client creates a webhook and receives events for all executions for selected 
workflows. 
{  
  name: web,
  active: true,
  events: [  ]
  “workflows”: [wf1, wf2] 
  url: http://example.com/webhook;,  
}

Opinions: 

DZ: my opinion: 
Option 1 for it is simple and convenient for a client. 
It seems like an optimal solution for “tracking executions and tasks” use case.

Option 2 is an overkill: makes it harder for a client (post workflow, post 
webhook, post execution, delete workflow, delete webhook) introduces lifecycle 
management problems (e.g., workflow deleted - webhook orphaned).

I vaguely recall someone from Heat compared these options and regretted one of 
them for security reasons, but can’t remember details.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] how to make Mistral build after keystone-pythonclient

[Dmitri Zimine]

Once we got dependencies on keystone-python client, Mistral doesn’t build for 
me on Mac. 

Before, I installed a new openssl (1.0.1h) - keystone authentication didn’t 
work with out it, remember? 

enykeev suggested to return to the old stock openssl, it worked.

but this sort of sucks to switch on and off? 

Ideas? 


DZ

/usr/bin/clang -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -g 
/Users/dzimine/Dev/openstack/mistral/.tox/py27/build/cryptography/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.o
 -o 
/Users/dzimine/Dev/openstack/mistral/.tox/py27/build/cryptography/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.so

running build_ext

building '_Cryptography_cffi_8f86901cxc1767c5a' extension

/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch 
x86_64 -g -O2 -DNDEBUG -g -O3 
-I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c 
cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_8f86901cxc1767c5a.c
 -o 
/Users/dzimine/Dev/openstack/mistral/.tox/py27/build/cryptography/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_8f86901cxc1767c5a.o

/usr/bin/clang -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -g 
/Users/dzimine/Dev/openstack/mistral/.tox/py27/build/cryptography/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_8f86901cxc1767c5a.o
 -o 
/Users/dzimine/Dev/openstack/mistral/.tox/py27/build/cryptography/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_8f86901cxc1767c5a.so

running build_ext

building '_Cryptography_cffi_4ed9e37dx4000d087' extension

creating 
/Users/dzimine/Dev/openstack/mistral/.tox/py27/build/cryptography/cryptography/hazmat/bindings/__pycache__/cryptography

creating 
/Users/dzimine/Dev/openstack/mistral/.tox/py27/build/cryptography/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat

creating 
/Users/dzimine/Dev/openstack/mistral/.tox/py27/build/cryptography/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat/bindings

creating 
/Users/dzimine/Dev/openstack/mistral/.tox/py27/build/cryptography/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat/bindings/__pycache__

/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch 
x86_64 -g -O2 -DNDEBUG -g -O3 
-I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c 
cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_4ed9e37dx4000d087.c 
-o 
/Users/dzimine/Dev/openstack/mistral/.tox/py27/build/cryptography/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_4ed9e37dx4000d087.o

/usr/bin/clang -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -g 
/Users/dzimine/Dev/openstack/mistral/.tox/py27/build/cryptography/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_4ed9e37dx4000d087.o
 -lcrypto -lssl -o 
/Users/dzimine/Dev/openstack/mistral/.tox/py27/build/cryptography/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_4ed9e37dx4000d087.so

Traceback (most recent call last):

  File string, line 16, in module

  File 
/Users/dzimine/Dev/openstack/mistral/.tox/py27/build/cryptography/setup.py, 
line 174, in module

test: PyTest,

  File 
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py,
 line 152, in setup

dist.run_commands()

  File 
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py,
 line 953, in run_commands

self.run_command(cmd)

  File 
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py,
 line 972, in run_command

cmd_obj.run()

  File string, line 14, in replacement_run

  File build/bdist.macosx-10.6-intel/egg/setuptools/command/egg_info.py, line 
261, in find_sources

  File build/bdist.macosx-10.6-intel/egg/setuptools/command/egg_info.py, line 
327, in run

  File build/bdist.macosx-10.6-intel/egg/setuptools/command/egg_info.py, line 
363, in add_defaults

  File build/bdist.macosx-10.6-intel/egg/setuptools/command/sdist.py, line 
219, in add_defaults

  File 
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py,
 line 312, in get_finalized_command

cmd_obj.ensure_finalized()

  File 
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py,
 line 109, in ensure_finalized

self.finalize_options()

  File build/bdist.macosx-10.6-intel/egg/setuptools/command/build_py.py, line 
73, in finalize_options

  File 
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_py.py,
 line 46, in finalize_options

('force', 'force'))

  File 
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py,
 line 298, in set_undefined_options

[openstack-dev] [mistral] Mistral API v2 - discussion

[Dmitri Zimine]

Hi Stackers, 

we are discussing the API, v2. 

The core team captured the initial thoughts 

here (most recent): 
https://etherpad.openstack.org/p/mistral-API-v2-discussion

and here: 
https://docs.google.com/a/stackstorm.com/document/d/12j66DZiyJahdnV8zXM_fCRJb0qxpsVS_eyhZvdHlIVU/edit

Have a look, leave your comments, questions, suggestions. 
On Monday’s IRC meeting we’ll talk more and finalize the draft. 

Thanks, 

DZ 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] Mistral milestone 0.1 blueprint review

[Dmitri Zimine]

Renat, team: 

The current plan of record on 0.1 here, 
https://launchpad.net/mistral/+milestone/0.1

Some suggested adjustments: 
https://docs.google.com/a/stackstorm.com/document/d/1Ukd6SDk9tMpn8kpti-kXaOzVhl8Jm1YWTh3Kw1A6lgQ/edit#

Pls use the doc to comment/disccuss and let’s decide by next mistral iRC 
meeting on Mon. 

Regards, Dmitri. ___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] Small questions re executor

[Dmitri Zimine]

Got some questions while fixing a bug https://review.openstack.org/#/c/102391/: 

* We must convey the action ERROR details back to the engine, and to the end 
user. Log is not sufficient. How exactly? Via context? Via extra parameters to 
convey_execution_results? Need a field in the model.
https://github.com/stackforge/mistral/blob/master/mistral/engine/drivers/default/executor.py#L46-L59


* What is the reason to update status on task failure in handle_task_error via 
direct DB access, not via convey_task_results? 
https://github.com/stackforge/mistral/blob/master/mistral/engine/drivers/default/executor.py#L61
 Bypassing convey_task_results can cause grief from missing TRACE statements to 
more serious stuff… And looks like we are failing the whole execution there? 
Just because one action had failed? Please clarify the intend here. Note: 
details may all go away while doing Refine Engine - Executor protocol 
blueprint 
https://blueprints.launchpad.net/mistral/+spec/mistral-engine-executor-protocol,
 we just need to clarify the intent

DZ ___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] Start task, and support for 'requires' / reverse workflow.

[Dmitri Zimine]

https://review.openstack.org/#/c/100390/

Angus asked: 
Why do we even need start: start-task?
Can't we parse the workbook and figure out what to run? Tasks at the bottom of 
the tree have no on-{fail,success} and then 
follow upwards until you find a task that is not referenced by anything.

Yes we can do this (patch almost ready). And It makes a perfect behavior for a 
direct flow [1]: all the top-level tasks begin to execute in parallel, and 
dependent tasks scheduled as dependencies resolve. No need to specify . 
Parallel execution is a default, unless explicitly constrained by 
on-success/on-error/on-finish. 

But for Reversed flow [2] it’s a BIG change of behavior. Instead of running 
only a subflow to satisfy a target task, it will run all the tasks (respecting 
dependencies). This is not practical. 

Eventually we want to support both direct and reverse as two types of workflow, 
without mixing the two syntaxes within the same workflow. This will require big 
refactoring (already planned). On the Atlanta summit we agreed to temporarily 
drop reversed workflow and re-introduce it later. [3] The start task in the API 
is a big pain for a direct flow. 

How should we proceed? Options I can think of: 

* Drop ‘reverse flow’ right now, stabilize ‘direct flow', reintroduce ‘reverse 
flow’ later (next cycle).
This will give us a clean final API on a good set of functionality, [almost] 
immediately. 

* Introduce two workflow types and correspondent API changes/additions on the 
current code base, before refactoring. 

* Put it off till the engine refactoring, factor the requirement of supporting 
two modes into the refactoring. It will be easy to do, but takes the longest. 

Other options? Opinions? Requests?

DZ  


[1] Direct flow has dependencies expressed by on-success/on-error/on-finish 
keyword on an upstream task
[2] Reverse flow is where dependencies are expressed by requires keyword on a 
downstream, dependent task
[3] https://etherpad.openstack.org/p/mistral-post-POC-questions

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] Community meeting reminder - 06/02/2014

[Dmitri Zimine]

Hi,

This is a reminder about the community meeting in IRC June 02 at 16.00 UTC at 
#openstack-meeting.

Agenda:
Review action items
Current status (quickly by team members)
Further plans
Open discussion

You can also find it at https://wiki.openstack.org/wiki/Meetings/MistralAgenda 
as well as links to the previous meeting minutes and logs.

- Dmitri Zimine___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] Mistral roadmap notes, from Atlanta summit (rough)

[Dmitri Zimine]

We shared and discussed the directions on Mistral development session,
and work through the list of smaller post POC steps, placing them on the 
roadmap.

The notes are here: https://etherpad.openstack.org/p/juno-summit-mistral.
Renat and I developed shared understanding on most of them. 

Next steps: create the blueprints, prioritize, implement. 

DZ.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Cleaning up configuration settings

[Dmitri Zimine]

+1 for breaking down the configuration by modules. 

Not sure about names for configuration group. Do you mean just using the same 
group name? or more? 

IMO groups are project specific; it doesn’t always make sense to use the same 
group name in the context of different projects. Our requirement is 1) 
auto-generate mistral.conf.example from the config.py and 2) sections make 
sense in the product context.

For example: how do we deal with rpc_backend and transport_url for oslo 
messaging? Should we do something like CONF.import(_transport_opts, 
“oslo.messaging.transport”, “transport”)? And use it by passing the group, not 
entire contfig, like:

transport = messaging.get_transport(cfg.messaging.CONF)
instead of
transport = messaging.get_transport(cfg.CONF)

DZ 


On May 16, 2014, at 12:46 PM, W Chan m4d.co...@gmail.com wrote:

 Regarding config opts for keystone, the keystoneclient middleware already 
 registers the opts at 
 https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/middleware/auth_token.py#L325
  under a keystone_authtoken group in the config file.  Currently, Mistral 
 registers the opts again at 
 https://github.com/stackforge/mistral/blob/master/mistral/config.py#L108 
 under a different configuration group.  Should we remove the duplicate from 
 Mistral and refactor the reference to keystone configurations to the 
 keystone_authtoken group?  This seems more consistent.
 
 
 On Thu, May 15, 2014 at 1:13 PM, W Chan m4d.co...@gmail.com wrote:
 Currently, the various configurations are registered in ./mistral/config.py.  
 The configurations are registered when mistral.config is referenced.  Given 
 the way the code is written, PEP8 throws referenced but not used error if 
 mistral.config is referenced but not called in the module.  In various use 
 cases, this is avoided by using importutils to import mistral.config (i.e. 
 https://github.com/stackforge/mistral/blob/master/mistral/tests/unit/engine/test_transport.py#L34).
   I want to break down registration code in ./mistral/config.py into separate 
 functions for api, engine, db, etc and move the registration closer to the 
 module where the configuration is needed.  Any objections?
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Search] Search project - update Atlanta'14

[Dmitri Zimine]

Hi folks, 

Last summit we proposed Search project for OpenStack[1]. We didn't get much 
traction and other things took priorities, so the project didn't take off 
during Havana cycle. 

Now - before and during the summit - more people expressed keen interest.  Some 
of us met on the summit and decided to unshelf the OpenStack Search. 

Let’s use this thread to get the interested parties together and define the 
next steps. 

The details for unshelfing: 

* Wiki - basic project description, and 2 min video showing Search in action.
  https://wiki.openstack.org/wiki/SearchProject

* Etherpad - captures Summit discussions:
  https://etherpad.openstack.org/p/search

* Github - the very rough prototype:
   * Search backend: https://github.com/StackStorm/search
   * Horizon plugin: https://github.com/StackStorm/search-horizon

* Discussion on the openstack-dev: 
   
http://lists.openstack.org/pipermail/openstack-dev/2013-November/019742.html

Regards, Dmitri. ___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral][TaskFlow] Integration plan

[Dmitri Zimine]

Renat, Joshua, Dmitri and Timur discussed the integration path for Mistral and 
TaskFlow. 
Quick summary - guys please correct any mistakes or omissions:

Taskflow - Mistral integration plan
1) Mistral: define the set of workflow primitives, define correspondent DSL
- Driven by user requirements. Keep to minimal. Implement in Mistral.
2) Taskflow: break out flow scheduler (decision maker) and persistence
- it'll be a low level library interference, Joshua has details
- once done, ready to re-prototype integration, iterate until happy
3) Move the primitives to TaskFlow (both teams)

4) Integrate TaskFlow into Mistral (replace Mistral workflow engine with 
TaskFlow engine)

Once integration complete, Mistral DSL will continue to work, all TaskFlow and 
Mistral workflows will be interchangeable, the code reused.

DZ. 

PS. When do we do it? We didn't commit to a date, 
but IMO (DZ) we start now, and with some luck get integrated in K cycle. 



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] Why keystone token validation fails in PyCharm, and how to fix it

[Dmitri Zimine]

The problem: Keystone authentication on MAC works from command line, but fails 
when run from PyCharm. And I want to run PyCharm to debug! 

The root cause: Keystone uses openssl cms which is only available in new 
openssl.  I installed openssl via brew, and got it in /usr/local/bin. Terminal 
works just fine because I have paths properly setup in /etc/paths file. But 
PyCharm and other non-terminal applications dont use it! 
 
The right solution: 
 create a file /etc/launchd.conf with the following line:
 setenv PATH /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin
   
The hack solution is:
 sudo mv /usr/bin/openssl /usr/bin/openssl_bak
 sudo ln -s /usr/bin/local/openssl /usr/bin/openssl

DZ 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary

[Dmitri Zimine]

Ivan, 

any update/progress on this? 

Thanks, DZ.


On Apr 15, 2014, at 10:21 AM, Joshua Harlow harlo...@yahoo-inc.com wrote:

 Well Ivan afaik is thinking through it, but being a community project its not 
 exactly easy to put estimations or timelines on things (I don't control Ivan, 
 or others).
 
 Likely faster if u guys want to get involved.
 
 -Josh
 
 From: Renat Akhmerov rakhme...@mirantis.com
 Reply-To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Date: Tuesday, April 15, 2014 at 12:35 AM
 To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Subject: Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow 
 integration summary
 
 
 On 15 Apr 2014, at 11:13, Joshua Harlow harlo...@yahoo-inc.com wrote:
 
 Sure, its not the fully complete lazy_engine, but piece by piece we can get 
 there.
 
 Did you make any estimations when it could happen? :)
 
 Of course code/contributions are welcome, as such things will benefit more 
 than just mistral, but openstack as a whole :-)
 
 OK.
 
 From: Kirill Izotov enyk...@stackstorm.com
 The whole idea of sub-flows within the scope of direct conditional 
 transitions is a bit unclear to me (and probably us all) at the moment, 
 though I'm trying to rely on them only as a means to lesser the complexity.
 
 Yes, eventually it’s for reducing complexity. I would just add that it opens 
 wide range of opportunities like:
 ability to combine multiple physically independent workflows
 reusability (using one workflow as a part of another)
 isolation (different namespaces for data flow contexts etc)
 
 Renat Akhmerov
 @ Mirantis Inc.
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary

[Dmitri Zimine]

We prototyped Mistral / TaskFlow integration and have a follow-up discussions. 

SUMMARY: Mistral (Workflow Service) can embed TaskFlow as a workflow library, 
with some required modifications to function resliently as a service, and for 
smooth integration. However, the TaskFlow flow controls are insufficient for 
Mistral use cases. 

Details discussed on other thirds. 
The prototype scope - [0]; code and discussion - [1] and techical highlights - 
[2].

DETAILS: 

1) Embedding TaskFlow inside Mistral:
* Required: make the engine lazy [3], [4].This is required to support 
long-running delegates and not loose tasks when engine manager process restarts.

* Persistence: need clarity how to replace or mix-in TaskFlow persistence with 
Mistral persistence. Renat is taking a look.

* Declaring Flows in YAML DSL: done for simplest flow. Need to prototype for 
data flow. Rich flow controls are missing in TaskFlow for a representative 
prototype.

* ActionRunners vs Taskflow Workers - not prototyped. Not a risk: both Mistral 
and TaskFlow implementations work. But we shall resolve the overlap. 

* Ignored for now - unlikely any risks: Keystone integration, Mistral event 
scheduler, Mistral declarative services and action definition.

2) TaskFlow library features
* Must: flow control - conditional transitions, references, expression 
evaluation, to express real-life workflows [5]. The required flow control 
primitives are 1) repeater 2) flow in flow 3) direct transition 4) conditional 
transition 5) multiple data. TaskFlow has 1) and 2), need to add 3/4/5. 

* Other details and smaller requests are in the discussion [1]

3) Next Steps proposed:
* Mistal team: summarize the requirements discussed and agreed on [2] and [3]
* Mistral team: code sample (tests?) on how Mistral would like to consume 
TaskFlow lazy engine 
* Taskflow team: Provide a design for alternative TaskExecutor approach 
(prototypes, boxes, arrows, crayons :)) 
* Decide on lazy engine
* Move the discussion on other elements on integration.

References:
[0] The scope of the prototype: 
https://etherpad.openstack.org/p/mistral-taskflow-prototype
[1] Prototype code and discussion https://github.com/enykeev/mistral/pull/1
[2] Techical summary 
http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html
[3] Email discussion on TaskFlow lazy eninge 
http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html
[4] IRC discussion Mistral/Taskflow http://paste.openstack.org/show/75389/
[5] Use cases https://github.com/dzimine/mistral-workflows/tree/add-usecases___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral][Taskflow] meet on IRC to talk over TaskFlow/Mistral integration

[Dmitri Zimine]

IRC to discuss http://tinyurl.com/k3s2gmy

Joshua, 2000 UTC doesn't quite work for Renat and Kirill (3 am their time). 

The overlap is: 
PST (UTC-7) UTC NOVT (UTC+7)

04pm (16:00)11pm (23:00)6am (06:00)
10pm (22:00)05am (05:00)12pm (12:00)

Kirill's pref is 3am UTC, early is ok, if needed. 
@Joshua can you do 3 am UTC (8 pm local?)
@Renat? 

Can we pencil 3:00 UTC on #openstack-mistral and adjust for Renat if needed?

DZ 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] Next crack at real workflows

[Dmitri Zimine]

Two more workflows drafted - cloud cron, and lifecycle, version 1. 

The mindset questions are: 
1) is DSL syntax expressive, and capable and convenient to handle real use 
cases?
2) most importantly: what are the implied workflow capabilities which make it 
all work? 
 
* Take a look here: 
https://github.com/dzimine/mistral-workflows/tree/add-usecases

* Leave your comments - generally, or  line-by-line, in the pull request  
https://github.com/dzimine/mistral-workflows/pull/1/files

* Fork, do your own modifications and do another pull request. 

* Or just reply with your comments  in email (lazy option :)) 

NOTE: please keep this thread for specific comments on DSL and workflow 
capabilities, create another thread if changing topic. Thanks! 

DZ ___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] How Mistral handling long running delegate tasks

[Dmitri Zimine]

Even more responses inline :)
On Mar 31, 2014, at 8:44 PM, Joshua Harlow harlo...@yahoo-inc.com wrote:

 More responses inline :)
 
 From: Dmitri Zimine d...@stackstorm.com
 Date: Monday, March 31, 2014 at 5:59 PM
 To: Joshua Harlow harlo...@yahoo-inc.com
 Cc: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Subject: Re: [openstack-dev] [Mistral] How Mistral handling long running 
 delegate tasks
 
 Inline...
 On Mar 27, 2014, at 5:10 PM, Joshua Harlow harlo...@yahoo-inc.com wrote:
 
 Thanks for the description!
 
 The steps here seem very much like what a taskflow engine does (which is 
 good).
 
 To connect this to how I think could work in taskflow.
 Someone creates tasks/flows describing the work to-be-done (converting a 
 DSL - taskflow tasks/flows/retry[1] objects…)
 On execute(workflow) engine creates a new workflow execution, computes the 
 first batch of tasks, creates executor for those tasks (remote, local…) and 
 executes those tasks.
 Waits for response back from futures returned from executor.
 Receives futures responses (or receives new response DELAY, for example), 
 or exceptions…
 Continues sending out batches of tasks that can be still be executing (aka 
 tasks that don't have dependency on output of delayed tasks).
 If any delayed tasks after repeating #2-5 as many times as it can, the 
 engine will shut itself down (see http://tinyurl.com/l3x3rrb).
 Why would engine treat long running tasks differently? The model Mistral 
 tried out is the engine sends the batch of tasks and goes asleep; the 
 'worker/executor' is calling engine back when the task(s) complete. Can it 
 be applied 
 
 
 Not all ways of using taskflow I think want to go 'asleep', some active 
 conductors I would think actually stay 'awake', that’s why I was thinking 
 this would be a comprimise, the usage of 'DELAY' (via exception or return) 
 would allow a task to say it is going to be finished sometime in the future 
 (btw started this @ https://review.openstack.org/#/c/84277/ --- WIP, likely 
 won't go into taskflow 0.2 but seems like a reasonable start on something 
 like this proposal). To me offering just the going asleep way means that all 
 users of taskflow need to have some type of entrypoint (rest, mq, other…) 
 that restarts the engine on result finishing. I'd rather not force that model 
 onto all users (since that seems inappropriate).
I understand and agree with not enforcing this model on all users. On a flip 
side, TaskFlow enforces the current model on Mistral :) It calls for supporting 
more models (to your earlier point of 3rd mode of operations). The BP proposal 
is a step in the direction but still a compromise. How can we support both 
'active' and 'passive' models, sharing the same 'library' - of all things 
responsible for control flow and data flow?

Here is a strawman (DISCLAIMER: this is sharing ideas and brainstorming: I 
don't know enough of TaksFlow to suggest): 
1) break the run loop: add engine.run_async() method which instead of looping 
the tasks till the flow completes (http://tinyurl.com/nou2em9), only schedules 
the first batch and returns; 
https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/graph_action.py#L61-L68

2) introduce engine.task_complete(…) method that marks the current task 
complete, computes the next tasks ready for execution, schedules them and 
returns; loosely, doing this part: 
https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/graph_action.py#L75-L85

3) make executor signal engine back by calling engine.task_complete(…): 
executor.execute_task() submits the task to the worker queue and calls 
engine.task_complete(status, results, etc.) instead of waiting for task 
completion and returning the results. 

And looks like it all can be done as a third lazy_engine, without messing the 
current two engines? 


 
 
 On delay task finishing some API/webhook/other (the mechanism imho 
 shouldn't be tied to webhooks, at least not in taskflow, but should be left 
 up to the user of taskflow to decide how to accomplish this) will be/must 
 be responsible for resuming the engine and setting the result for the 
 previous delayed task.
 Oh no, webhook is the way to expose it to 3rd party system. From the library 
 standpoint it's just an API call. 
 
 One can do it even now by getting the appropriate Flow_details, 
 instantiating and engine (flow, flow_details) and running it to continue 
 from where it left out. Is it how you mean it? But I keep on dreaming of a 
 passive version of TaskFlow engine which treats all tasks the same and 
 exposes one extra method - handle_tasks. 
 
 
 I was thinking that, although I'm unsure on the one extra method idea, can u 
 explain more :) What would handle_tasks do? Restart/resume the engine 
 (basically the work u stated 'getting the appropriate Flow_details, 
 instantiating and engine (flow, flow_details) and running

Re: [openstack-dev] [Mistral][TaskFlow] Long running actions

[Dmitri Zimine]

On Apr 1, 2014, at 3:43 AM, Renat Akhmerov rakhme...@mirantis.com wrote:
 On 25 Mar 2014, at 01:51, Joshua Harlow harlo...@yahoo-inc.com wrote:
 
 The first execution model I would call the local execution model, this model 
 involves forming tasks and flows and then executing them inside an 
 application, that application is running for the duration of the workflow 
 (although if it crashes it can re-establish the task and flows that it was 
 doing and attempt to resume them). This could also be what openstack 
 projects would call the 'conductor' approach where nova, ironic, trove have 
 a conductor which manages these long-running actions (the conductor is 
 alive/running throughout the duration of these workflows, although it may be 
 restarted while running). The restarting + resuming part is something that 
 openstack hasn't handled so gracefully currently, typically requiring either 
 some type of cleanup at restart (or by operations), with taskflow using this 
 model the resumption part makes it possible to resume from the last saved 
 state (this connects into the persistence model that taskflow uses, the 
 state transitions, how execution occurrs itself...). 
 
 The second execution model is an extension of the first, whereby there is 
 still a type of 'conductor' that is managing the life-time of the workflow, 
 but instead of locally executing tasks in the conductor itself tasks are now 
 executed on remote-workers (see http://tinyurl.com/lf3yqe4
 ). The engine currently still is 'alive' for the life-time of the execution, 
 although the work that it is doing is relatively minimal (since its not 
 actually executing any task code, but proxying those requests to others 
 works). The engine while running does the conducting of the remote-workers 
 (saving persistence details, doing state-transtions, getting results, 
 sending requests to workers…).
 
 These two execution models are special cases of what you call “lazy execution 
 model” (or passive as we call it). To illustrate this idea we can take a look 
 at the first sequence diagram at [0], we basically will see the following 
 interaction:
 
 1) engine --(task)-- queue --(task)-- worker
 2) execute task
 3) worker --(result)-- queue --(result)-- engine
 
 This is how TaskFlow worker based model works.
 
 If we loosen the requirement in 3) and assume that not only worker can send a 
 task result back to engine we’ll get our passive model. Instead of worker it 
 can be anything else (some external system) that knows how to make this call. 
 A particular way is not too important, it can be a direct message or it can 
 be hidden behind an API method. In Mistral it’s now a REST API method however 
 we’re about to decouple engine from REST API so that engine is a standalone 
 process and listens to a queue. So worker-based model is basically the same 
 with the only strict requirement that only worker sends a result back.
 
 In order to implement local execution model on top of “lazy execution model” 
 we just need to abstract a transport (queue) so that we can use an in-process 
 transport. That’s it. It’s what Mistral already has implemented. Again, we 
 see that “lazy execution model” is more universal.
 
 IMO this “lazy execution model” should be the main execution model that 
 TaskFlow supports, others can be easily implemented on top of it. But the 
 opposite assertion is wrong. IMO this is the most important obstacle in all 
 our discussions, the reason why we don’t always understand each other well 
 enough. I know it may be a lot of work to shift a paradigm in TaskFlow team 
 but if we did that we would get enough freedom for using TaskFlow in lots of 
 cases.
 
 Let me know what you think. I might have missed something.

DZ: Interesting idea! So that other models of execution are based on lazy 
execution model? TaskFlow implements this, we can use it, and for other clients 
more convenient higher level execution models are provided? Interesting. Makes 
sense.
@Joshua? @Kirill? Others? 

 
 === HA ===
 
 So this is an interesting question, and to me is strongly connected to how 
 your engines are executing (and the persistence and state-transitions that 
 they go through while running). Without persistence of state and transitions 
 there is no good way (a bad way of course can be created, by just redoing 
 all the work, but that's not always feasible or the best option) to 
 accomplish resuming in a sane manner and there is also imho no way to 
 accomplish any type of automated HA of workflows. 
 
 Sure, no questions here.
 
 Let me describe:
 
 When you save the states of a workflow and any intermediate results of a 
 workflow to some database (for example) and the engine (see above models) 
 which is being used (for example the conductor type from above) the 
 application containing that engine may be prone to crashes (or just being 
 powered off due to software upgrades...). Since taskflows key primitives 
 were made to allow for 

Re: [openstack-dev] [Mistral] Engine overview and proposal

[Dmitri Zimine]

I am in the full agreement with the proposed approach (risked to copy below, 
let's see if email handles your diagrams): 

* Single Engine handles multiple executions asynchronously, in non-blocking way.
* Persistence access only from API and/or Engine, Engine writes, API reads. 
* Action runes don't talk to DB - it's very simple protocol and queue is 
perfectly fine. 
* This is scalable and as fault tolerant as underlying DB and Queue. Engine 
restart is no loss of info with right persistense; ActionRunner restart  is a 
lost of Action, which can fail for 100 other reasons thus expected, and with 
the right retry policy potentially recoverable. 

I'll inline minor points in the etherpad. 



 API EngineQueueDatabaseWorkers
  |   |   | | | ||   |
--- +-+  |   | | | ||   |
 |S|  |   | | | ||   |
 |t| -- +-+  | | | ||   |
 |a| | |  - - - - - - - - ||   |
 |r| | | - - - - t - - - - - - - - -  +-+  |
 |t| -- +-+  | | | |   | |  |
--- +-+  |   | | | |   | |  |
  |   |  +-+ - r - - - - - - - - - -  +-+  |
--- +-+  |  | | - - - - - -  R|   |
 |I|  |  | | - - t - - - - - - - - -  +-+  |
 |n|  |  | | - - t - - - - - - - - - - -  +-+
 |f|  |  +-+| | |   | | | |
 |o| - - - - - - - - - - - -  |   | | | |
--- +-+  |  +-+ - r - - - - - - - - - - -+-+ | |
  |   |  | | - - - - - -  R|  | |
--- +-+  |  +-+| | ||  | |
 |S| -- +-+  | | | ||  | |
 |t| | | - - - - - - - -  ||  | |
 |o| -- +-+  | | | ||  | |
 |p|  |  +-+ - r  - - - - - - - - - - - - +-+
--- +-+  |  | | - - - - - -  R|   |
  |   |  |%|| | ||   |
  |   |  +-+| | ||   |
  |   |   | | | ||   |


On Mar 31, 2014, at 4:09 AM, Kirill Izotov enyk...@stackstorm.com wrote:

 I have an idea regarding engine design i want to share with you. But first, 
 it seems like we need a small overview of the current implementations.
 
 I'm not sure how ML will react on a bunch of ASCII graphs, so here is an 
 etherpad: 
 https://etherpad.openstack.org/p/mistral-engine-overview-and-proposal
 
 What do you think, guys, is this the way to go?
 
 -- 
 Kirill Izotov
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] How Mistral handling long running delegate tasks

[Dmitri Zimine]

Inline...
On Mar 27, 2014, at 5:10 PM, Joshua Harlow harlo...@yahoo-inc.com wrote:

 Thanks for the description!
 
 The steps here seem very much like what a taskflow engine does (which is 
 good).
 
 To connect this to how I think could work in taskflow.
 Someone creates tasks/flows describing the work to-be-done (converting a DSL 
 - taskflow tasks/flows/retry[1] objects…)
 On execute(workflow) engine creates a new workflow execution, computes the 
 first batch of tasks, creates executor for those tasks (remote, local…) and 
 executes those tasks.
 Waits for response back from futures returned from executor.
 Receives futures responses (or receives new response DELAY, for example), or 
 exceptions…
 Continues sending out batches of tasks that can be still be executing (aka 
 tasks that don't have dependency on output of delayed tasks).
 If any delayed tasks after repeating #2-5 as many times as it can, the engine 
 will shut itself down (see http://tinyurl.com/l3x3rrb).
Why would engine treat long running tasks differently? The model Mistral tried 
out is the engine sends the batch of tasks and goes asleep; the 
'worker/executor' is calling engine back when the task(s) complete. Can it be 
applied 
 On delay task finishing some API/webhook/other (the mechanism imho shouldn't 
 be tied to webhooks, at least not in taskflow, but should be left up to the 
 user of taskflow to decide how to accomplish this) will be/must be 
 responsible for resuming the engine and setting the result for the previous 
 delayed task.
Oh no, webhook is the way to expose it to 3rd party system. From the library 
standpoint it's just an API call. 

One can do it even now by getting the appropriate Flow_details, instantiating 
and engine (flow, flow_details) and running it to continue from where it left 
out. Is it how you mean it? But I keep on dreaming of a passive version of 
TaskFlow engine which treats all tasks the same and exposes one extra method - 
handle_tasks. 
 Repeat 2 - 7 until all tasks have executed/failed.
 Profit!

 This seems like it could be accomplished, although there are race conditions 
 in the #6 (what if multiple delayed requests are received at the same time)? 
 What locking is done to ensure that this doesn't cause conflicts?
Engine must handle concurrent calls of mutation methods - start, stop, 
handle_action. How -  differs depending on engine running in multiple threads 
or in event loop on queues of calls. 

 Does the POC solve that part (no simultaneous step #5 from below)?
Yes although we may want to revisit the current solution. 

 There was a mention of a watch-dog (ideally to ensure that delayed tasks 
 can't just sit around forever), was that implemented?
If _delayed_ tasks and 'normal' tasks are treat alike, this is just a matter of 
timeout as a generic property on a task. So Mistral didn't have to have it. For 
the proposal above, a separate treatment is necessary for _delayed_ tasks. 
 
 [1] https://wiki.openstack.org/wiki/TaskFlow#Retries (new feature!)
This is nice. I would call it a 'repeater': running a sub flow several times 
with various data for various reasons is reacher then 'retry'. 
What about the 'retry policy' on individual task? 

 
 From: Dmitri Zimine d...@stackstorm.com
 Reply-To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Date: Thursday, March 27, 2014 at 4:43 PM
 To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Subject: [openstack-dev] [Mistral] How Mistral handling long running delegate 
 tasks
 
 Following up on http://tinyurl.com/l8gtmsw and http://tinyurl.com/n3v9lt8: 
 this explains how Mistral handles long running delegate tasks. Note that a 
 'passive' workflow engine can handle both normal tasks and delegates the 
 same way. I'll also put that on ActionDesign wiki, after discussion.
 
 Diagram: 
 https://docs.google.com/a/stackstorm.com/drawings/d/147_EpdatpN_sOLQ0LS07SWhaC3N85c95TkKMAeQ_a4c/edit?usp=sharing
 
 1. On start(workflow), engine creates a new workflow execution, computes the 
 first batch of tasks, sends them to ActionRunner [1].
 2. ActionRunner creates an action and calls action.run(input)
 3. Action does the work (compute (10!)), produce the results,  and return 
 the results to executor. If it returns, status=SUCCESS. If it fails it 
 throws exception, status=ERROR.
 4. ActionRunner notifies Engine that the task is complete 
 task_done(execution, task, status, results)[2]
 5. Engine computes the next task(s) ready to trigger, according to control 
 flow and data flow, and sends them to ActionRunner.
 6. Like step 2: ActionRunner calls the action's run(input)
 7. A delegate action doesn't produce results: it calls out the 3rd party 
 system, which is expected to make a callback to a workflow service with the 
 results. It returns to ActionRunner without results, immediately.  
 8. ActionRunner marks status=RUNNING [?]
 9. 3rd party system

[openstack-dev] [Mistral] Community meeting reminder - 03/30/2014

[Dmitri Zimine]

Hi,

This is a reminder that we’ll have a community meeting today as usually at 
16.00 UTC at #openstack-meeting.

Here’s the agenda (also at 
https://wiki.openstack.org/wiki/Meetings/MistralAgenda):
Review action items
Current status (quickly by team members)
POC scope and readiness
Mistral on top of TaskFlow prototype - summary, next steps
Open discussion

Looking forward to see you there.

Dmitri Zimine
@ StackStorm

PS. Hope Renat is coming back this week. ___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] How Mistral handling long running delegate tasks

[Dmitri Zimine]

Following up on http://tinyurl.com/l8gtmsw and http://tinyurl.com/n3v9lt8:  
this explains how Mistral handles long running delegate tasks. Note that a 
'passive' workflow engine can handle both normal tasks and delegates the same 
way. I'll also put that on ActionDesign wiki, after discussion.

Diagram: 
https://docs.google.com/a/stackstorm.com/drawings/d/147_EpdatpN_sOLQ0LS07SWhaC3N85c95TkKMAeQ_a4c/edit?usp=sharing

1. On start(workflow), engine creates a new workflow execution, computes the 
first batch of tasks, sends them to ActionRunner [1].
2. ActionRunner creates an action and calls action.run(input)
3. Action does the work (compute (10!)), produce the results,  and return the 
results to executor. If it returns, status=SUCCESS. If it fails it throws 
exception, status=ERROR.
4. ActionRunner notifies Engine that the task is complete task_done(execution, 
task, status, results)[2]
5. Engine computes the next task(s) ready to trigger, according to control flow 
and data flow, and sends them to ActionRunner.
6. Like step 2: ActionRunner calls the action's run(input)
7. A delegate action doesn't produce results: it calls out the 3rd party 
system, which is expected to make a callback to a workflow service with the 
results. It returns to ActionRunner without results, immediately.  
8. ActionRunner marks status=RUNNING [?]
9. 3rd party system takes 'long time' == longer then any system component can 
be assumed to stay alive. 
10. 3rd party component calls Mistral WebHook which resolves to 
engine.task_complete(workbook, id, status, results)  

Comments: 
* One Engine handles multiple executions of multiple workflows. It exposes two 
main operations: start(workflow) and task_complete(execution, task, status, 
results), and responsible for defining the next batch of tasks based on control 
flow and data flow. Engine is passive - it runs in a hosts' thread. Engine and 
ActionRunner communicate via task queues asynchronously, for details, see  
https://wiki.openstack.org/wiki/Mistral/POC 

* Engine doesn't distinct sync and async actions, it doesn't deal with Actions 
at all. It only reacts to task completions, handling the results, updating the 
state, and queuing next set of tasks.

* Only Action can know and define if it is a delegate or not. Some protocol 
required to let ActionRunner know that the action is not returning the results 
immediately. A convention of returning None may be sufficient. 

* Mistral exposes  engine.task_done in the REST API so 3rd party systems can 
call a web hook.

DZ.

[1]  I use ActionRunner instead of Executor (current name) to avoid confusion: 
it is Engine which is responsible for executions, and ActionRunner only runs 
actions. We should rename it in the code.

[2] I use task_done for briefly and out of pure spite, in the code it is 
conveny_task_results.___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral][TaskFlow] Long running actions

[Dmitri Zimine]

 connected to how 
 your engines are executing (and the persistence and state-transitions that 
 they go through while running). Without persistence of state and transitions 
 there is no good way (a bad way of course can be created, by just redoing all 
 the work, but that's not always feasible or the best option) to accomplish 
 resuming in a sane manner and there is also imho no way to accomplish any 
 type of automated HA of workflows. Since taskflow was concieved to manage the 
 states and transitions of tasks and flows it gains the ability to do this 
 resuming but it also gains the ability to automatically provide execution HA 
 to its users.
 
 Let me describe:
 
 When you save the states of a workflow and any intermediate results of a 
 workflow to some database (for example) and the engine (see above models) 
 which is being used (for example the conductor type from above) the 
 application containing that engine may be prone to crashes (or just being 
 powered off due to software upgrades...). Since taskflows key primitives were 
 made to allow for resuming when a crash occurs, it is relatively simple to 
 allow another application (also running a conductor) to resume whatever that 
 prior application was doing when it crashed. Now most users of taskflow don't 
 want to have to do this resumption manually (although they can if they want) 
 so it would be expected that the other versions of that application would be 
 running would automatically 'know' how to 'take-over' the work of the failed 
 application. This is where the concept of the taskflows 'jobboard' 
 (http://tinyurl.com/klg358j) comes into play, where a jobboard can be backed 
 by something like zookeeper (which provides notifications of lock 
 lose/release to others automatically). The jobboard is the place where the 
 other applications would be looking to 'take-over' the other failed 
 applications work (by using zookeeper 'atomic' primitives designed for this 
 type of usage) and they would also be releasing the work back for others to 
 'take-over' when there own zookeeper connection is lost (zookeeper handles 
 this this natively).
 
 --
 
 Now as for how much of mistral would change from the above, I don't know, but 
 thats why it's a POC.
 
 -Josh
 
 From: Joshua Harlow harlo...@yahoo-inc.com
 Date: Friday, March 21, 2014 at 1:14 PM
 To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Cc: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Subject: Re: [openstack-dev] [Mistral][TaskFlow] Long running actions
 
 Will advise soon, out sick with not so fun case of poison oak, will reply 
 next week (hopefully) when I'm less incapacitated...
 
 Sent from my really tiny device...
 
 On Mar 21, 2014, at 3:24 AM, Renat Akhmerov rakhme...@mirantis.com wrote:
 
 Valid concerns. It would be great to get Joshua involved in this 
 discussion. If it’s possible to do in TaskFlow he could advise on how 
 exactly.
 
 Renat Akhmerov
 @ Mirantis Inc.
 
 
 
 On 21 Mar 2014, at 16:23, Stan Lagun sla...@mirantis.com wrote:
 
 Don't forget HA issues. Mistral can be restarted at any moment and need to 
 be able to proceed from the place it was interrupted on another instance. 
 In theory it can be addressed by TaskFlow but I'm not sure it can be done 
 without complete redesign of it
 
 
 On Fri, Mar 21, 2014 at 8:33 AM, W Chan m4d.co...@gmail.com wrote:
 Can the long running task be handled by putting the target task in the 
 workflow in a persisted state until either an event triggers it or 
 timeout occurs?  An event (human approval or trigger from an external 
 system) sent to the transport will rejuvenate the task.  The timeout is 
 configurable by the end user up to a certain time limit set by the 
 mistral admin.  
 
 Based on the TaskFlow examples, it seems like the engine instance 
 managing the workflow will be in memory until the flow is completed.  
 Unless there's other options to schedule tasks in TaskFlow, if we have 
 too many of these workflows with long running tasks, seems like it'll 
 become a memory issue for mistral...
 
 
 On Thu, Mar 20, 2014 at 3:07 PM, Dmitri Zimine d...@stackstorm.com 
 wrote:
 
 For the 'asynchronous manner' discussion see 
 http://tinyurl.com/n3v9lt8; I'm still not sure why u would want to make 
 is_sync/is_async a primitive concept in a workflow system, shouldn't 
 this be only up to the entity running the workflow to decide? Why is a 
 task allowed to be sync/async, that has major side-effects for 
 state-persistence, resumption (and to me is a incorrect abstraction to 
 provide) and general workflow execution control, I'd be very careful 
 with this (which is why I am hesitant to add it without much much more 
 discussion).
 
 
 Let's remove the confusion caused by async. All tasks [may] run async 
 from the engine standpoint, agreed. 
 
 Long running tasks - that's it.
 
 Examples: wait_5_days, run_hadoop_job

Re: [openstack-dev] [Mistral] task update at end of handle_task in executor

[Dmitri Zimine]

My understanding is: it's the engine which finalizes the task results, based on 
the status returned by the task via convey_task_result call. 

https://github.com/stackforge/mistral/blob/master/mistral/engine/abstract_engine.py#L82-L84
https://github.com/stackforge/mistral/blob/master/mistral/engine/scalable/executor/server.py#L44-L66

In case of async tasks, executor keeps the task status at RUNNING, and a 3rd 
party system will call convey_task_resutls on engine. 

 https://github.com/stackforge/mistral/blob/master/mistral/engine/scalable/executor/server.py#L123,

This line however looks like a bug to me:  at best it doesnt do much and at 
worst it overwrites the ERROR previously set in here http://tinyurl.com/q5lps2h 

Nicolay, any better explanation?


DZ 

On Mar 26, 2014, at 6:20 PM, W Chan m4d.co...@gmail.com wrote:

 Regarding 
 https://github.com/stackforge/mistral/blob/master/mistral/engine/scalable/executor/server.py#L123,
  should the status be set to SUCCESS instead of RUNNING?  If not, can someone 
 clarify why the task should remain RUNNING?
 
 Thanks.
 Winson
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] Community meeting minutes - 03/24/2014

[Dmitri Zimine]

Hi folks, thanks for taking part in Mistral meeting on #openstack-meeting

Here is the minutes and the full log.

Minutes 
http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-03-24-16.03.html

Log: 
http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-03-24-16.03.log.html

Join us next time March 31, at the same time 

(note that with summer time in effect now it's 9am Pacific)

Cheers, DZ. ___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral][TaskFlow] Long running actions

[Dmitri Zimine]

 For the 'asynchronous manner' discussion see http://tinyurl.com/n3v9lt8; I'm 
 still not sure why u would want to make is_sync/is_async a primitive concept 
 in a workflow system, shouldn't this be only up to the entity running the 
 workflow to decide? Why is a task allowed to be sync/async, that has major 
 side-effects for state-persistence, resumption (and to me is a incorrect 
 abstraction to provide) and general workflow execution control, I'd be very 
 careful with this (which is why I am hesitant to add it without much much 
 more discussion).


Let's remove the confusion caused by async. All tasks [may] run async from 
the engine standpoint, agreed. 

Long running tasks - that's it.

Examples: wait_5_days, run_hadoop_job, take_human_input. 
The Task doesn't do the job: it delegates to an external system. The flow 
execution needs to wait  (5 days passed, hadoob job finished with data x, user 
inputs y), and than continue with the received results.

The requirement is to survive a restart of any WF component without loosing the 
state of the long running operation.

Does TaskFlow already have a way to do it? Or ongoing ideas, considerations? If 
yes let's review. Else let's brainstorm together. 

I agree,
 that has major side-effects for state-persistence, resumption (and to me is a 
 incorrect abstraction to provide) and general workflow execution control, I'd 
 be very careful with this

But these requirement  comes from customers'  use cases: wait_5_day - lifecycle 
management workflow, long running external system - Murano requirements, user 
input - workflow for operation automations with control gate checks, provisions 
which require 'approval' steps, etc. 

DZ 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack][Mistral] Adding new core reviewers

[Dmitri Zimine]

+1 for Nikolay - he's indeed the most involved in all aspect.
and yes I am up to doing it. 

DZ.

On Mar 19, 2014, at 4:35 AM, Renat Akhmerov rakhme...@mirantis.com wrote:

 Team,
 
 So far I’ve been just the only one core member of the team. I started feeling 
 lonely :) Since the project team and the project itself has now grown (thanks 
 to StackStorm and Intel) I think it’s time to think about extending the core 
 team.
 
 I would propose:
 Nikolay Makhotkin (nmakhotkin at launchpad). He's been working on the project 
 since almost the very beginning and made significant contribution (design, 
 reviews, code).
 Dmitri Zimine (i-dz at launchpad). Dmitri joined the project about 2 months 
 ago. Since then he’s made a series of important high-quality commits, a lot 
 of valuable reviews and, IMO most importantly, he has a solid vision of the 
 project in general (requirements, use cases, comparison to other 
 technologies) and has a pro-active viewpoint in all our discussions.
 
 Thoughts?
 
 Renat Akhmerov
 @ Mirantis Inc.
 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Actions design BP

[Dmitri Zimine]

Thanks Renat for a clear design summary, 
thanks Joshua for the questions, 
+1 to let's move TaskFlow vs Mistral discussion to separate thread, 
and my questions/comments on 
https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign below: 

- Async actions: how do results of async action communicates back? 
My understanding it it assumes that the remote system will call back to mistral 
with action execution id, it's on the engine to call-back, and action needs to 
let the engine know to expect call-back. Let's put the explanation here. 

- is_sync() - consider using an attribute instead -  @mistral.async
- can we think of a way to unify sync and async actions from engine's 
standpoint? So that we don't special-case it in the engine? @ Joshua - does 
something similar exists in TaskFlow already?

- def dry_run() - maybe name test, let's stress that this method should 
return a representative sample output. 

- Input - need a facility to declare, validate and list input parameters. Like 
VALID_KEYS=['url', 'parameters''] , def validate(): 

- class HTTPAction(object):
def __init__(self, url, params, method, headers, body):
Not happy about declaring parameters explicitly. How about using * args 
**kvargs, or 'parameters' dictionary? 

- DSL In-Place Declaration - I did minor edits in the section, please check. 

DZ. 
- 
On Mar 12, 2014, at 6:54 PM, Joshua Harlow harlo...@yahoo-inc.com wrote:

 So taskflow has tasks, which seems comparable to actions?
 
 I guess I should get tired of asking but why recreate the same stuff ;)
 
 The questions listed:
 
 - Does action need to have revert() method along with run() method?
 - How does action expose errors occurring during it's work?
 
 - In what form does action return a result?
 
 
 And more @ https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign
 
 And quite a few others that haven't been mentioned (how does a action
 retry? How does a action report partial progress? What's the
 intertask/state persistence mechanism?) have been worked on by the
 taskflow team for a while now...
 
 https://github.com/openstack/taskflow/blob/master/taskflow/task.py#L31
 (and others...)
 
 Anyways, I know mistral is still POC/pilot/prototype... but seems like
 more duplicate worked that could just be avoided ;)
 
 -Josh
 
 -Original Message-
 From: Renat Akhmerov rakhme...@mirantis.com
 Reply-To: OpenStack Development Mailing List (not for usage questions)
 openstack-dev@lists.openstack.org
 Date: Tuesday, March 11, 2014 at 11:32 PM
 To: OpenStack Development Mailing List (not for usage questions)
 openstack-dev@lists.openstack.org
 Subject: [openstack-dev] [Mistral] Actions design BP
 
 Team,
 
 I started summarizing all the thoughts and ideas that we¹ve been
 discussing for a while regarding actions. The main driver for this work
 is that the system keeps evolving and we still don¹t have a comprehensive
 understanding of that part. Additionally, we keep getting a lot of
 requests and questions from our potential users which are related to
 actions (Œwill they be extensible?¹, Œwill they have dry-run feature?¹,
 Œwhat are the ways to configure and group them?¹ and so on and so forth).
 So although we¹re still in a Pilot phase we need to start this work in
 parallel. Even now lack of solid understanding of it creates a lot of
 problems in pilot development.
 
 I created a BP at launchpad [0] which has a reference to detailed
 specification [1]. It¹s still in progress but you could already leave
 your early feedback so that I don¹t go in a wrong direction too far.
 
 The highest priority now is still finishing the pilot so we shouldn¹t
 start implementing everything described in BP right now. However, some of
 the things have to be adjusted asap (like Action interface and the main
 implementation principles).
 
 [0]: 
 https://blueprints.launchpad.net/mistral/+spec/mistral-actions-design
 [1]: https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign
 
 Renat Akhmerov
 @ Mirantis Inc.
 
 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Local vs. Scalable Engine

[Dmitri Zimine]

We have access to all configuration parameters in the context of api.py. May be 
you don't pass it but just instantiate it where you need it? Or I may 
misunderstand what you're trying to do...

DZ 

PS: can you generate and update mistral.config.example to include new oslo 
messaging options? I forgot to mention it on review on time. 


On Mar 13, 2014, at 11:15 AM, W Chan m4d.co...@gmail.com wrote:

 On the transport variable, the problem I see isn't with passing the variable 
 to the engine and executor.  It's passing the transport into the API layer.  
 The API layer is a pecan app and I currently don't see a way where the 
 transport variable can be passed to it directly.  I'm looking at 
 https://github.com/stackforge/mistral/blob/master/mistral/cmd/api.py#L50 and 
 https://github.com/stackforge/mistral/blob/master/mistral/api/app.py#L44.  Do 
 you have any suggestion?  Thanks. 
 
 
 On Thu, Mar 13, 2014 at 1:30 AM, Renat Akhmerov rakhme...@mirantis.com 
 wrote:
 
 On 13 Mar 2014, at 10:40, W Chan m4d.co...@gmail.com wrote:
 
 I can write a method in base test to start local executor.  I will do that 
 as a separate bp.  
 Ok.
 
 After the engine is made standalone, the API will communicate to the engine 
 and the engine to the executor via the oslo.messaging transport.  This means 
 that for the local option, we need to start all three components (API, 
 engine, and executor) on the same process.  If the long term goal as you 
 stated above is to use separate launchers for these components, this means 
 that the API launcher needs to duplicate all the logic to launch the engine 
 and the executor. Hence, my proposal here is to move the logic to launch the 
 components into a common module and either have a single generic launch 
 script that launch specific components based on the CLI options or have 
 separate launch scripts that reference the appropriate launch function from 
 the common module.
 
 Ok, I see your point. Then I would suggest we have one script which we could 
 use to run all the components (any subset of of them). So for those 
 components we specified when launching the script we use this local 
 transport. Btw, scheduler eventually should become a standalone component 
 too, so we have 4 components.
 
 The RPC client/server in oslo.messaging do not determine the transport.  The 
 transport is determine via oslo.config and then given explicitly to the RPC 
 client/server.  
 https://github.com/stackforge/mistral/blob/master/mistral/engine/scalable/engine.py#L31
  and 
 https://github.com/stackforge/mistral/blob/master/mistral/cmd/task_executor.py#L63
  are examples for the client and server respectively.  The in process Queue 
 is instantiated within this transport object from the fake driver.  For the 
 local option, all three components need to share the same transport in 
 order to have the Queue in scope. Thus, we will need some method to have 
 this transport object visible to all three components and hence my proposal 
 to use a global variable and a factory method. 
 I’m still not sure I follow your point here.. Looking at the links you 
 provided I see this:
 
 transport = messaging.get_transport(cfg.CONF)
 
 So my point here is we can make this call once in the launching script and 
 pass it to engine/executor (and now API too if we want it to be launched by 
 the same script). Of course, we’ll have to change the way how we initialize 
 these components, but I believe we can do it. So it’s just a dependency 
 injection. And in this case we wouldn’t need to use a global variable. Am I 
 still missing something?
 
 
 Renat Akhmerov
 @ Mirantis Inc.
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Crack at a Real life workflow

[Dmitri Zimine]

?
 
 $0.02
 
 -S
 
 
 
 On 03/05/2014 10:50 PM, Dmitri Zimine wrote:
 Folks, 
 
 I took a crack at using our DSL to build a real-world workflow. 
 Just to see how it feels to write it. And how it compares with
 alternative tools. 
 
 This one automates a page from OpenStack operation
 guide: 
 http://docs.openstack.org/trunk/openstack-ops/content/maintenance.html#planned_maintenance_compute_node
  
 
 Here it is https://gist.github.com/dzimine/9380941
 or here http://paste.openstack.org/show/72741/
 
 I have a bunch of comments, implicit assumptions, and questions which
 came to mind while writing it. Want your and other people's opinions on 
 it. 
 
 But gist and paste don't let annotate lines!!! :(
 
 May be we can put it on the review board, even with no intention to
 check in,  to use for discussion? 
 
 Any interest?
 
 DZ 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Crack at a Real life workflow

[Dmitri Zimine]

I just moved the sample to Git; let's leverage git review for specific comments 
on the syntax. 

https://github.com/dzimine/mistral-workflows/commit/d8c4a8c845e9ca49f6ea94362cef60489f2a46a3

DZ 

On Mar 6, 2014, at 10:36 PM, Dmitri Zimine d...@stackstorm.com wrote:

 Folks, thanks for the input! 
 
 @Joe: 
 
 Hopefully Renat covered the differences.  Yet I am interested in how the same 
 workflow can be expressed as Salt state(s) or Ansible playbooks. Can you (or 
 someone else who knows them well) take a stub? 
 
 
 @Joshua
 I am still new to Mistral and learning, but I think it _is_ relevant to 
 taskflow. Should we meet, and you help me catch up? Thanks! 
 
 @Sandy:
 Aaahr, I used the D word?!  :) I keep on arguing that YAML workflow 
 representation doesn't make DSL. 
 
 And YES to the object model first to define the workflow, with 
 YAML/JSON/PythonDSL/what-else as a syntax to build it. We are having these 
 discussions on another thread and reviews. 
 
 Basically, in order to make a grammar expressive enough to work across a
 web interface, we essentially end up writing a crappy language. Instead,
 we should focus on the callback hooks to something higher level to deal
 with these issues. Minstral should just say I'm done this task, what
 should I do next? and the callback service can make decisions on where
 in the graph to go next.
 
 There must be some misunderstanding. Mistral _does follow AWS / BPEL engines 
 approach, it is both doing I'm done this task, what should I do next? 
 (executor) and callback service (engine that coordinates the flow and keeps 
 the state). Like decider and activity workers in AWS Simple Workflow.
 
 Engine maintains the state. Executors run tasks. Object model describes 
 workflow as a graph of tasks with transitions, conditions, etc. YAML is one 
 way to define a workflow. Nothing controversial :) 
 
 @all:
 
 Wether one writes Python code or uses yaml? Depends on the user. There are 
 good arguments for YAML. But if it's crappy, it looses. We want to see how it 
 feels to write it. To me, mixed feelings so far, but promising. What do you 
 guys think?
 
 Comments welcome here: 
 https://github.com/dzimine/mistral-workflows/commit/d8c4a8c845e9ca49f6ea94362cef60489f2a46a3
 
 
 DZ 
 
 
 On Mar 6, 2014, at 10:41 AM, Sandy Walsh sandy.wa...@rackspace.com wrote:
 
 
 
 On 03/06/2014 02:16 PM, Renat Akhmerov wrote:
 IMO, it looks not bad (sorry, I’m biased too) even now. Keep in mind this 
 is not the final version, we keep making it more expressive and concise.
 
 As for killer object model it’s not 100% clear what you mean. As always, 
 devil in the details. This is a web service with all the consequences. I 
 assume what you call “object model” here is nothing else but a python 
 binding for the web service which we’re also working on. Custom python 
 logic you mentioned will also be possible to easily integrate. Like I said, 
 it’s still a pilot stage of the project.
 
 Yeah, the REST aspect is where the tricky part comes in :)
 
 Basically, in order to make a grammar expressive enough to work across a
 web interface, we essentially end up writing a crappy language. Instead,
 we should focus on the callback hooks to something higher level to deal
 with these issues. Minstral should just say I'm done this task, what
 should I do next? and the callback service can make decisions on where
 in the graph to go next.
 
 Likewise with things like sending emails from the backend. Minstral
 should just call a webhook and let the receiver deal with active
 states as they choose.
 
 Which is why modelling this stuff in code is usually always better and
 why I'd lean towards the TaskFlow approach to the problem. They're
 tackling this from a library perspective first and then (possibly)
 turning it into a service. Just seems like a better fit. It's also the
 approach taken by Amazon Simple Workflow and many BPEL engines.
 
 -S
 
 
 Renat Akhmerov
 @ Mirantis Inc.
 
 
 
 On 06 Mar 2014, at 22:26, Joshua Harlow harlo...@yahoo-inc.com wrote:
 
 That sounds a little similar to what taskflow is trying to do (I am of 
 course biased).
 
 I agree with letting the native language implement the basics 
 (expressions, assignment...) and then building the domain ontop of that. 
 Just seems more natural IMHO, and is similar to what linq (in c#) has done.
 
 My 3 cents.
 
 Sent from my really tiny device...
 
 On Mar 6, 2014, at 5:33 AM, Sandy Walsh sandy.wa...@rackspace.com 
 wrote:
 
 DSL's are tricky beasts. On one hand I like giving a tool to
 non-developers so they can do their jobs, but I always cringe when the
 DSL reinvents the wheel for basic stuff (compound assignment
 expressions, conditionals, etc).
 
 YAML isn't really a DSL per se, in the sense that it has no language
 constructs. As compared to a Ruby-based DSL (for example) where you
 still have Ruby under the hood for the basic stuff and extensions to the
 language for the domain-specific stuff.
 
 Honestly, I'd like

[openstack-dev] [Mistral] Crack at a Real life workflow

[Dmitri Zimine]

Folks, 

I took a crack at using our DSL to build a real-world workflow. 
Just to see how it feels to write it. And how it compares with alternative 
tools. 

This one automates a page from OpenStack operation guide: 
http://docs.openstack.org/trunk/openstack-ops/content/maintenance.html#planned_maintenance_compute_node
 

Here it is https://gist.github.com/dzimine/9380941
or here http://paste.openstack.org/show/72741/

I have a bunch of comments, implicit assumptions, and questions which came to 
mind while writing it. Want your and other people's opinions on it. 

But gist and paste don't let annotate lines!!! :(

May be we can put it on the review board, even with no intention to check in,  
to use for discussion? 

Any interest?

DZ ___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Renaming events to triggers and move them out of Workflow

[Dmitri Zimine]

Agree on both. 


On Feb 26, 2014, at 8:51 PM, Renat Akhmerov rakhme...@mirantis.com wrote:

 Hi team,
 
 When I tell peopleI about Mistral I always have a hard time explaining why we 
 use term “event” for declaring ways to start workflows. For example, take a 
 look at the following snippet:
 
 Workflow:
   ...
   events:
 execute_backup:
type: periodic
tasks: execute_backup
parameters:
cron-pattern: */1 * * * *
 
 Here we just tell Mistral “we should run backup workflow every minute”.
 
 My suggestion is to rename “events” to “triggers” because actually events are 
 going to be sent by Mistral when we implement notification mechanism (sending 
 events about over HTTP or AMQP about changes in workflows' and tasks’ state).
 
 I would also suggest we move events out of “Workflow” section since it’s not 
 actually a part of workflow.
 
 Thoughts?
 
 If you agree I’ll create a blueprint for this.
 
 Renat Akhmerov
 @ Mirantis Inc.
 
 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Understanding parameters for tasks and actions

[Dmitri Zimine]

My understanding, correct me if I'm missing the intention: 

Action are defined as code, aka REST_API or SEND_EMAIL. 
These are base actions, they are analogous to function definitions. 

Tasks is a set of parameters for the action.

Actions can be also defined declaratively, under Services, based on base 
actions. 
These are service, or declared actions, they are analogous to partials. 
In the simplest form, they 1) set some of the parameters on the base functions 
and 2) define outputs.

But I guess what Renat is trying to capture here, is to also define new input 
parameters, (analogous to  changing a function signature). If so, they become 
adapters; and there need to be a) define new parameters and b) transform the 
new parameters (input) into original parameters of the base action. 

Is this how we think about it? If yes, the syntax to express it might be 
something like this

# Snippet 1.
Services:
  Nova:
type: REST_API
  parameters:
baseUrl: $.novaURL
actions:
  createVM:
parameters: # partial set for the parameters of base action.
   method: POST
   url: /servers/{{service-id}} # sets up the initial parameter, 
using the new input 
input: 
   service-id: # defines a new input
output: 
   select: $.server.id
   store-as: vm_id
# Snippet 2.
Workflow:
  tasks:
createVM:
  action: Nova:createVM
parameters:
  service-id: {{$.vm_id}}
on-success: waitForIP
on-error: sendCreateVMError

Better ideas on referencing input, better than {{service-id}}are welcome. 

The point here is we don't refer the context variables in Service definitions. 
Only transformation of input, with some static text. The service actions now 
look symmetric on input/output. 

As for output: I would keep it as is. It defines the transformation of the 
results of base action: a) defining new output variables and b) providing their 
values, by transforming the output of base task. Very symmetric to 'input'. 
Currently we stores into the global context, but it doesn't have to be so; I 
think we eventually (soon) need to have task-specific section in the context 
(more on this later)

Thoughts? 


DZ.

PS. For the record, I am split on declarative action definitions. I like it, 
but only as long as it stays simple. Very soon it becomes easier to express it 
in code then in YAML. 


On Feb 25, 2014, at 11:34 PM, Renat Akhmerov rakhme...@mirantis.com wrote:

 Hi team,
 
 I’m currently working on the first version of Data Flow and I would like to 
 make sure we all clearly understand how to interpret “parameters for tasks 
 and actions when we declare them in Mistral DSL. I feel like I’m getting lost 
 here a little bit. The problem is that we still don’t have a solid DSL spec 
 since we keep changing our vision (especially after new members joined the 
 team). But that may be fine, it’s life.
 
 I also have a couple of suggestions that I’d like to discuss with you. Sorry 
 if that seems verbose, I’ll try to be as concise as possible.
 
 I took a couple of snippets from [1] and put them in here.
 
 # Snippet 1.
 Services:
   Nova:
 type: REST_API
 parameters:
   baseUrl: $.novaURL
 actions:
   createVM:
 parameters:
   url: /servers/{$.vm_id}
   method: POST
 output:
   select: $.server.id
   store-as: vm_id
 
 # Snippet 2.
 Workflow:
   tasks:
 createVM:
   action: Nova:createVM
   on-success: waitForIP
   on-error: sendCreateVMError
 
 “$.” - handle to workflow execution storage (what we call ‘context’ now) 
 where we keep workflow variables.
 
 Let’s say our workflow input is JSON like this:
 {
   “novaURL”: “http://localhost:123”,
   “image_id”: “123
 }
 
 Questions
 
 So the things that I don’t like or am not sure about:
 
 1. Task “createVM” needs to use “image_id” but it doesn’t have any 
 information about it its declaration.
 According to the current vision it should be like 
 createVM:
   action: Nova:createVM
   parameters:
 image_id: $.image_id
 And at runtime “image_id should be resolved to “123” get passed to action 
 and, in fact, be kind of the third parameter along with “url” and “method”. 
 This is specifically interesting because on one hand we have different types 
 of parameters: “url” and “method” for REST_API action define the nature of 
 the action itself. But “image_id” on the other hand is a dynamic data coming 
 eventually from user input.
 So the question is: do we need to differentiate between these types of 
 parameters explicitly and make a part of the specification?
 We also had a notion of “task-parameters” for action declarations which is 
 supposed to be used to declare this second type of parameters (dynamic) but 
 do we really need it? I guess if we clearly declare input and output at task 
 level then actions should be able to use 

Re: [openstack-dev] [Mistral] Renaming action types

[Dmitri Zimine]

+1

Should we use VERB_NOUN pattern? Or relax it for some obvious? I can't figure a 
good VERB_NOUN for HTTP. REQUEST_HTTP is dull. 

BTW it's interesting that we assume that a service can contain only actions of 
the same type. 

DZ 

On Feb 26, 2014, at 1:10 AM, Nikolay Makhotkin nmakhot...@mirantis.com wrote:

 Agree, I don't see any thing that makes sense with words 'REST' and 'API' too.
 
 
 On Wed, Feb 26, 2014 at 11:38 AM, Renat Akhmerov rakhme...@mirantis.com 
 wrote:
 Folks,
 
 I’m proposing to rename these two action types REST_API and MISTRAL_REST_API 
 to HTTP and MISTRAL_HTTP. Words “REST” and “API” don’t look correct to me, if 
 you look at
 
 Services:
   Nova:
 type: REST_API
 parameters:
   baseUrl: {$.novaURL}
 actions:
   createVM:
 parameters:
   url: /servers/{$.vm_id}
   method: POST

 There’s no information about “REST” or “API” here. It’s just a spec how to 
 form an HTTP request.
 
 Thoughts?
 
 Renat Akhmerov
 @ Mirantis Inc.
 
 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
 -- 
 Best Regards,
 Nikolay
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Defining term DSL

[Dmitri Zimine]

We do use the term DSL, I invite you guys to clarify, how exactly. 

Based on the terminology from [1], it's not part of the model, but the language 
that describes the model in the file. And theoretically this may be not the 
only language to express the workflow. Once the file is parsed, we operate on 
model, not on the language. 

I am afraid we are breaking an abstraction when begin to call things 
DSLWorkbook or DSLWorkflow. What is the difference between Workbook and 
DSLWorkbook, and how DSL is relevant here? 

[1] https://wiki.openstack.org/wiki/Mistral, 

DZ 
On Feb 26, 2014, at 7:19 AM, Renat Akhmerov rakhme...@mirantis.com wrote:

 I don’t see any issues with term DSL (Domain Specific Language). This is 
 really a language which 'workbook definitions’ are written in.
 
 Dmitri, could you please provide more details on why you question it?
 
 Thanks
 
 Renat Akhmerov
 @ Mirantis Inc.
  
 
 On 26 Feb 2014, at 20:12, Nikolay Makhotkin nmakhot...@mirantis.com wrote:
 
 Due to the comment to https://review.openstack.org/#/c/75888/1 there is a 
 quiestion: 
 
 Do we use term DSL or something else? 
 I think the word 'DSL' is more fit thing that we call 'workbook definition', 
 some text describing workflows, services, tasks and actions. And processing 
 module for this also has name 'dsl'.
 
 Thoughts? Dmitri?
 
 Nikolay,
 Mirantis Inc.
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] What is the model?

[Dmitri Zimine]

As a learning exercise, I was trying to extract the intend behind the DSL and 
current code. Giving that many things are still moving, I filled up the blanks 
with imagination. Here is my understanding on where you are going: 

https://docs.google.com/a/stackstorm.com/drawings/d/1qBxrmQ8F7YFlz930zEbHKUxgzc3ty7JrlmoTR5TxQlo/edit

Is it where we are going? 
Can we use it to bring us the new team members to your understanding? 
Please comment here or right on the document. 

Nikolay's recent refactoring seems to be going in this direction : 
https://review.openstack.org/#/c/75888/  

Note: I am not making any points here (yet), I am trying to understand the 
intention. 

Ok, except One point: 
We are missing a separation of ActionSpec and ActionData. They have distinct 
roles. First  - action spec, from Service/actions section - defines the new 
action declaratively from existing action. Second - action data from 
Workflow/tasks/task - defines parameters for a particular action instance.  

DZ



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral]

[Dmitri Zimine]

I have created a blueprint to capture the intention to simplify calling 
standard actions:

https://blueprints.launchpad.net/mistral/+spec/mistral-shorthand-action-in-dsl

DZ 

On Feb 11, 2014, at 7:40 AM, Dmitri Zimine d...@stackstorm.com wrote:

 Yes it makes sense, let's draft how it may look;
 
 and also think over implementation implications - now we separate task 
 parameters, action parameters, and service parameters, we may need to merge 
 them when instantiating the action. 
 
 DZ. 
 
 On Feb 11, 2014, at 6:19 AM, Renat Akhmerov rakhme...@mirantis.com wrote:
 
 Dmitry, I think you are right here. I think for simple case we should be 
 able to use in-place action definition without having to define the action 
 separately. Like you said it’s only valuable if we need to reuse it.
 
 The only difference I see between std:send-email and something like REST_API 
 is that a set of parameters for the latter is dynamic (versus std:send-email 
 where it’s always “recipients”, “subject”, “body”). Even though it’s still 
 the same protocol (HTTP) but a particular request representation may be 
 different (i.e. query string, headers, the structure of body in case POST 
 etc.). But I think that doesn’t cancel the idea of being able to define the 
 action along with the task itself.
 
 So good point. As for the syntax itself, we need to think it over. In the 
 snippet you provided “action: std:REST_API”, so we need to make sure not to 
 have ambiguities in the ways how we can refer actions. A convention could be 
 “if we don’t use a namespace we assume that there’s a separate action 
 definition included into the same workbook, otherwise it should be 
 considered in-place action definition and task property “action” refers to 
 an action type rather than the action itself”. Does that make sense?
 
 
 Renat Akhmerov
 @ Mirantis Inc.
 
 On 11 Feb 2014, at 16:23, Dmitri Zimine d...@stackstorm.com wrote:
 
 Do we have (or think about) a shorthand to calling REST_API action, without 
 defining a service? 
 
 FULL  DSL:
 
 Services:
 TimeService:
 type: REST_API
 parameters:
   baseUrl:http://api.timezonedb.com
   key:my_api_key
 actions:
   get-time:
 task-parameters:
   zone:
 Workflow:
 tasks:
timeInToronto:
   action: TimeService:get-time
   parameters:
 zone: America/Toronto
 
 SHORTCUT - may look something like this: 
 
 Workflow:
 tasks:
 timeInToronto:
 action:std:REST_API
 parameters:
   baseUrl: http://api.timezonedb.com;
   method: GET
   parameters: zone=/America/Torontokey=my_api_key
   
 Why asking:  
 
 1) analogy with std:send-email action. I wonder do we have to make user 
 define Service for std:send-email? and I think that for standard tasks we 
 shouldn't have to. If there is any thinking on REST_API, it may apply here. 
 
 2) For a one-off web service calls the complete syntax is may be overkill 
 (but yes, it comes handy for reuse). See examples below. 
 
 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Local vs. Scalable Engine

[Dmitri Zimine]

I agree with Winson's points. Inline.

On Feb 24, 2014, at 8:31 PM, Renat Akhmerov rakhme...@mirantis.com wrote:

 
 On 25 Feb 2014, at 07:12, W Chan m4d.co...@gmail.com wrote:
 
 As I understand, the local engine runs the task immediately whereas the 
 scalable engine sends it over the message queue to one or more executors.  
 
 Correct.

Note: that local is confusing here, in process will reflect what it is 
doing better. 

 
 In what circumstances would we see a Mistral user using a local engine 
 (other than testing) instead of the scalable engine?
 
 Yes, mostly testing we it could be used for demonstration purposes also or in 
 the environments where installing RabbitMQ is not desirable.
 
 If we are keeping the local engine, can we move the abstraction to the 
 executor instead, having drivers for a local executor and remote executor?  
 The message flow from the engine to the executor would be consistent, it's 
 just where the request will be processed.  
 
 I think I get the idea and it sounds good to me. We could really have 
 executor in both cases but the transport from engine to executor can be 
 different. Is that what you’re suggesting? And what do you call driver here?

+1 to abstraction to the executor, indeed the local and remote engines today 
differ only by how they invoke executor, e.g. transport / driver.

 
 And since we are porting to oslo.messaging, there's already a fake driver 
 that allows for an in process Queue for local execution.  The local executor 
 can be a derivative of that fake driver for non-testing purposes.  And if we 
 don't want to use an in process queue here to avoid the complexity, we can 
 have the client side module of the executor determine whether to dispatch to 
 a local executor vs. RPC call to a remote executor.
 
 Yes, that sounds interesting. Could you please write up some etherpad with 
 details explaining your idea?
 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging

[Dmitri Zimine]

Winson, 

While you're looking into this and working on the design, may be also think 
through other executor/engine communications.

We talked about executor communicating to engine over 3 channels (DB, REST, 
RabbitMQ) which I wasn't happy about ;) and put it off for some time. May be it 
can be rationalized as part of your design. 

DZ. 

On Feb 24, 2014, at 11:21 AM, W Chan m4d.co...@gmail.com wrote:

 Renat,
 
 Regarding your comments on change https://review.openstack.org/#/c/75609/, I 
 don't think the port to oslo.messaging is just a swap from pika to 
 oslo.messaging.  OpenStack services as I understand is usually implemented as 
 an RPC client/server over a messaging transport.  Sync vs async calls are 
 done via the RPC client call and cast respectively.  The messaging transport 
 is abstracted and concrete implementation is done via drivers/plugins.  So 
 the architecture of the executor if ported to oslo.messaging needs to include 
 a client, a server, and a transport.  The consumer (in this case the mistral 
 engine) instantiates an instance of the client for the executor, makes the 
 method call to handle task, the client then sends the request over the 
 transport to the server.  The server picks up the request from the exchange 
 and processes the request.  If cast (async), the client side returns 
 immediately.  If call (sync), the client side waits for a response from the 
 server over a reply_q (a unique queue for the session in the transport).  
 Also, oslo.messaging allows versioning in the message. Major version change 
 indicates API contract changes.  Minor version indicates backend changes but 
 with API compatibility.  
 
 So, where I'm headed with this change...  I'm implementing the basic 
 structure/scaffolding for the new executor service using oslo.messaging 
 (default transport with rabbit).  Since the whole change will take a few 
 rounds, I don't want to disrupt any changes that the team is making at the 
 moment and so I'm building the structure separately.  I'm also adding 
 versioning (v1) in the module structure to anticipate any versioning changes 
 in the future.   I expect the change request will lead to some discussion as 
 we are doing here.  I will migrate the core operations of the executor 
 (handle_task, handle_task_error, do_task_action) to the server component when 
 we agree on the architecture and switch the consumer (engine) to use the new 
 RPC client for the executor instead of sending the message to the queue over 
 pika.  Also, the launcher for ./mistral/cmd/task_executor.py will change as 
 well in subsequent round.  An example launcher is here 
 https://github.com/uhobawuhot/interceptor/blob/master/bin/interceptor-engine. 
  The interceptor project here is what I use to research how oslo.messaging 
 works.  I hope this is clear. The blueprint only changes how the request and 
 response are being transported.  It shouldn't change how the executor 
 currently works.
 
 Finally, can you clarify the difference between local vs scalable engine?  I 
 personally do not prefer to explicitly name the engine scalable because this 
 requirement should be in the engine by default and we do not need to 
 explicitly state/separate that.  But if this is a roadblock for the change, I 
 can put the scalable structure back in the change to move this forward.
 
 Thanks.
 Winson
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] Notes on action YAML declaration and naming

[Dmitri Zimine]

We touched this on review https://review.openstack.org/#/c/73205/, and fixed a 
bit, bringing it up here to further discuss at slightly higher level. 

Let's go over a tiny bit of  YAML definition, clarifying terminology on the way.

Current DSL snippet: 
actions:
   my-action
  parameters:
  foo: bar
  response: # just agreed to change to 'results' 
  select: '$.server_id'  
  store_as: v1

In the code, we refer to action.result_helper

1) Note that response is not exactly a parameter. It doesn't doesn't refer to 
data. It's  (query, variable) pairs, that are used to parse the results and 
post data to global context [1]. The terms response, or result, do not reflect 
what is actually happening here. Suggestions? Save? Publish? Result Handler? 

2) Whichever name we use for this output transformer, shall it be under 
parameters?   

3) how do we call action/task parameters? Think 'model' (which reflects in 
yaml,  code, docs, talk, etc.)
   input and output? (+1)
   in and out? (-1)  
   request and response? (-1) good for WebServices but not generic enough
   parameters and results? (ok)

4) Syntax simplification: can we drop 'parameters' keyword? Anything under 
action is action parameters, unless it's a reserved keyword, which the 
implementation can parse out. 

actions:
   my-action
  foo: bar
  task-parameters: # not a keyword, specific to REST_API
  flavor_id:
  image_id:
  publish:  # keyword
  select: '$.server_id'  
  store_as: v1

[1] Save discussing the way we work with data flow, and talking alternatives, 
to the next time. 

DZ. 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Notes on action YAML declaration and naming

[Dmitri Zimine]

I like output, too. But it should go with 'input'
In summary, there are two alternatives. 
Note that I moved task-parameters under parameters. Ok with this?

actions:
   my-action
  input:
 foo: bar
 task-parameters: 
flavor_id:
image_id:
  output: 
  select: '$.server_id'  
  store_as: v1

this maps to action(input, *output)

actions:
   my-action
  parameters:
 foo: bar
 task-parameters: 
flavor_id:
image_id:
  result: 
  select: '$.server_id'  
  store_as: v1

this maps to result=action(parameters)


On Feb 14, 2014, at 8:40 AM, Renat Akhmerov rakhme...@mirantis.com wrote:

 “output” looks nice!
 
 
 Renat Akhmerov
 @ Mirantis Inc.
 
 On 14 Feb 2014, at 20:26, Nikolay Makhotkin nmakhot...@mirantis.com wrote:
 
 Current DSL snippet: 
 actions:
my-action
   parameters:
   foo: bar
   response: # just agreed to change to 'results' 
   select: '$.server_id'  
   store_as: v1
 
 'result' sounds better than 'response' and, I think, more fit to action 
 description.
 And I suggest for a new word - 'output'; actually, this block describe how 
 the output will be taken and stored.
 
 However, I agree that this block should be at action-property level:
 
 actions:
my-action
   result: 
  select: '$.server_id'  
  store_as: vm_id
   parameters:
  foo: bar
   
 
 
 On Fri, Feb 14, 2014 at 12:36 PM, Renat Akhmerov rakhme...@mirantis.com 
 wrote:
 
 On 14 Feb 2014, at 15:02, Dmitri Zimine d...@stackstorm.com wrote:
 
 Current DSL snippet: 
 actions:
my-action
   parameters:
   foo: bar
   response: # just agreed to change to 'results’ 
 
 Just a note: “response” indentation here is not correct, it’s not a 
 parameter called “response” but rather a property of “my-action”.
 
   select: '$.server_id'  
   store_as: v1
 
 In the code, we refer to action.result_helper
 
 1) Note that response is not exactly a parameter. It doesn't doesn't refer 
 to data. It's  (query, variable) pairs, that are used to parse the results 
 and post data to global context [1]. The terms response, or result, do not 
 reflect what is actually happening here. Suggestions? Save? Publish? Result 
 Handler? 
 
 For explicitness we can use something like “result-handler” and initially I 
 thought about this option. But I personally love conciseness and I think 
 name “result” would be ok for this section meaning it defines the structure 
 of the action result. “handler” is not 100% precise too because we don’t 
 actually handle a result here, we define the rules how to get this result.
 
 I would appreciate to see other suggestions though.
 
 2) Whichever name we use for this output transformer, shall it be under 
 parameters?
 
 No, what we have in this section is like a return value type for a regular 
 method. Parameters define action input.
 
 3) how do we call action/task parameters? Think 'model' (which reflects in 
 yaml,  code, docs, talk, etc.)
input and output? (+1)
in and out? (-1)  
request and response? (-1) good for WebServices but not generic enough
parameters and results? (ok)
 
 Could you please clarify your questions here? Not sure I’m following...
 
 4) Syntax simplification: can we drop 'parameters' keyword? Anything under 
 action is action parameters, unless it's a reserved keyword, which the 
 implementation can parse out. 
 
 actions:
my-action
   foo: bar
   task-parameters: # not a keyword, specific to REST_API
   flavor_id:
   image_id:
   publish:  # keyword
   select: '$.server_id'  
   store_as: v1
 
 It will create problems like name ambiguity in case we need to have a 
 parameter with the same names as keywords (“task-parameters” and “publish” 
 in your example). My preference would be to leave it explicit.
 
 Renat
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
 -- 
 Best Regards,
 Nikolay
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Notes on action YAML declaration and naming

[Dmitri Zimine]

Ok, I see.  

Do we have a spec that describes this?
Lets spell it out and describe the whole picture of input, output, parameters, 
and result. 

DZ 


On Feb 14, 2014, at 1:01 PM, Nikolay Makhotkin nmakhot...@mirantis.com wrote:

 Dmitri, in our concerns under word 'input' we assume a block contains the 
 info about how the input data will be taken for corresponding task from 
 initial context. So, it will be a kind of expression (e.g. YAQL). 
 
 Renat, am I right?
 
 
 On Fri, Feb 14, 2014 at 9:51 PM, Dmitri Zimine d...@stackstorm.com wrote:
 I like output, too. But it should go with 'input'
 In summary, there are two alternatives. 
 Note that I moved task-parameters under parameters. Ok with this?
 
 actions:
my-action
   input:
  foo: bar
  task-parameters: 
 flavor_id:
 image_id:
   output: 
   select: '$.server_id'  
   store_as: v1
 
 this maps to action(input, *output)
 
 actions:
my-action
   parameters:
  foo: bar
  task-parameters: 
 flavor_id:
 image_id:
   result: 
   select: '$.server_id'  
   store_as: v1
 
 this maps to result=action(parameters)
 
 
 On Feb 14, 2014, at 8:40 AM, Renat Akhmerov rakhme...@mirantis.com wrote:
 
 “output” looks nice!
 
 
 Renat Akhmerov
 @ Mirantis Inc.
 
 On 14 Feb 2014, at 20:26, Nikolay Makhotkin nmakhot...@mirantis.com wrote:
 
 Current DSL snippet: 
 actions:
my-action
   parameters:
   foo: bar
   response: # just agreed to change to 'results' 
   select: '$.server_id'  
   store_as: v1
 
 'result' sounds better than 'response' and, I think, more fit to action 
 description.
 And I suggest for a new word - 'output'; actually, this block describe how 
 the output will be taken and stored.
 
 However, I agree that this block should be at action-property level:
 
 actions:
my-action
   result: 
  select: '$.server_id'  
  store_as: vm_id
   parameters:
  foo: bar
   
 
 
 On Fri, Feb 14, 2014 at 12:36 PM, Renat Akhmerov rakhme...@mirantis.com 
 wrote:
 
 On 14 Feb 2014, at 15:02, Dmitri Zimine d...@stackstorm.com wrote:
 
 Current DSL snippet: 
 actions:
my-action
   parameters:
   foo: bar
   response: # just agreed to change to 'results’ 
 
 Just a note: “response” indentation here is not correct, it’s not a 
 parameter called “response” but rather a property of “my-action”.
 
   select: '$.server_id'  
   store_as: v1
 
 In the code, we refer to action.result_helper
 
 1) Note that response is not exactly a parameter. It doesn't doesn't refer 
 to data. It's  (query, variable) pairs, that are used to parse the results 
 and post data to global context [1]. The terms response, or result, do not 
 reflect what is actually happening here. Suggestions? Save? Publish? 
 Result Handler? 
 
 For explicitness we can use something like “result-handler” and initially I 
 thought about this option. But I personally love conciseness and I think 
 name “result” would be ok for this section meaning it defines the structure 
 of the action result. “handler” is not 100% precise too because we don’t 
 actually handle a result here, we define the rules how to get this result.
 
 I would appreciate to see other suggestions though.
 
 2) Whichever name we use for this output transformer, shall it be under 
 parameters?
 
 No, what we have in this section is like a return value type for a regular 
 method. Parameters define action input.
 
 3) how do we call action/task parameters? Think 'model' (which reflects in 
 yaml,  code, docs, talk, etc.)
input and output? (+1)
in and out? (-1)  
request and response? (-1) good for WebServices but not generic enough
parameters and results? (ok)
 
 Could you please clarify your questions here? Not sure I’m following...
 
 4) Syntax simplification: can we drop 'parameters' keyword? Anything under 
 action is action parameters, unless it's a reserved keyword, which the 
 implementation can parse out. 
 
 actions:
my-action
   foo: bar
   task-parameters: # not a keyword, specific to REST_API
   flavor_id:
   image_id:
   publish:  # keyword
   select: '$.server_id'  
   store_as: v1
 
 It will create problems like name ambiguity in case we need to have a 
 parameter with the same names as keywords (“task-parameters” and “publish” 
 in your example). My preference would be to leave it explicit.
 
 Renat
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
 -- 
 Best Regards,
 Nikolay
 ___
 OpenStack-dev mailing list
 OpenStack-dev

[openstack-dev] [Mistral]

[Dmitri Zimine]

Do we have (or think about) a shorthand to calling REST_API action, without 
defining a service? 

FULL  DSL:

Services:
  TimeService:
type: REST_API
parameters:
  baseUrl:http://api.timezonedb.com
  key:my_api_key
actions:
  get-time:
task-parameters:
  zone:
Workflow:
   tasks:
  timeInToronto:
 action: TimeService:get-time
 parameters:
   zone: America/Toronto
   
SHORTCUT - may look something like this: 

Workflow:
   tasks:
   timeInToronto:
action:std:REST_API
parameters:
  baseUrl: http://api.timezonedb.com;
  method: GET
  parameters: zone=/America/Torontokey=my_api_key
  
Why asking:  

1) analogy with std:send-email action. I wonder do we have to make user define 
Service for std:send-email? and I think that for standard tasks we shouldn't 
have to. If there is any thinking on REST_API, it may apply here. 

2) For a one-off web service calls the complete syntax is may be overkill (but 
yes, it comes handy for reuse). See examples below. 




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral]

[Dmitri Zimine]

Yes it makes sense, let's draft how it may look;

and also think over implementation implications - now we separate task 
parameters, action parameters, and service parameters, we may need to merge 
them when instantiating the action. 

DZ. 

On Feb 11, 2014, at 6:19 AM, Renat Akhmerov rakhme...@mirantis.com wrote:

 Dmitry, I think you are right here. I think for simple case we should be able 
 to use in-place action definition without having to define the action 
 separately. Like you said it’s only valuable if we need to reuse it.
 
 The only difference I see between std:send-email and something like REST_API 
 is that a set of parameters for the latter is dynamic (versus std:send-email 
 where it’s always “recipients”, “subject”, “body”). Even though it’s still 
 the same protocol (HTTP) but a particular request representation may be 
 different (i.e. query string, headers, the structure of body in case POST 
 etc.). But I think that doesn’t cancel the idea of being able to define the 
 action along with the task itself.
 
 So good point. As for the syntax itself, we need to think it over. In the 
 snippet you provided “action: std:REST_API”, so we need to make sure not to 
 have ambiguities in the ways how we can refer actions. A convention could be 
 “if we don’t use a namespace we assume that there’s a separate action 
 definition included into the same workbook, otherwise it should be considered 
 in-place action definition and task property “action” refers to an action 
 type rather than the action itself”. Does that make sense?

 
 Renat Akhmerov
 @ Mirantis Inc.
 
 On 11 Feb 2014, at 16:23, Dmitri Zimine d...@stackstorm.com wrote:
 
 Do we have (or think about) a shorthand to calling REST_API action, without 
 defining a service? 
 
 FULL  DSL:
 
 Services:
 TimeService:
  type: REST_API
  parameters:
baseUrl:http://api.timezonedb.com
key:my_api_key
  actions:
get-time:
  task-parameters:
zone:
 Workflow:
  tasks:
 timeInToronto:
action: TimeService:get-time
parameters:
  zone: America/Toronto
 
 SHORTCUT - may look something like this: 
 
 Workflow:
  tasks:
  timeInToronto:
  action:std:REST_API
  parameters:
baseUrl: http://api.timezonedb.com;
method: GET
parameters: zone=/America/Torontokey=my_api_key

 Why asking:  
 
 1) analogy with std:send-email action. I wonder do we have to make user 
 define Service for std:send-email? and I think that for standard tasks we 
 shouldn't have to. If there is any thinking on REST_API, it may apply here. 
 
 2) For a one-off web service calls the complete syntax is may be overkill 
 (but yes, it comes handy for reuse). See examples below. 
 
 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] Comments on DSL

[Dmitri Zimine]

Ops! missed [openstack-dev], fixing…

Following up from yesterday's community meeting

I am still catching up with the project,  still miss a lot of context. Put my 
questions and comments on DSL definition: 
https://etherpad.openstack.org/p/mistral-dsl-discussion

Let's review, and thanks for helping us get up to speed. 

DZ



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Mistral] Comments on DSL

[Dmitri Zimine]

Following up from yesterday's community meeting

I am still catching up with the project,  still miss a lot of context. Put my 
questions and comments on DSL definition: 
https://etherpad.openstack.org/p/mistral-dsl-discussion

Let's review, and thanks for helping us get up to speed. 

DZ ___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev