[openstack-dev] [Mistral] Event Subscription

[W Chan]

Regarding blueprint to register event listeners to notify client
applications on state changes (
https://blueprints.launchpad.net/mistral/+spec/mistral-event-listeners-http),
I want to propose the following.

1. Refer this feature as event subscription instead of callback
2. Event subscription supports only HTTP web hooks with retry policy in
this first attempt
3. Event subscription can be defined for a list of specific events,
workflows, projects, domains, or any combinations (all if list is empty).
4. Decorator to publish event (similar to logging) and place decorators at
Engine and TaskPolicy methods @
https://github.com/stackforge/mistral/blob/master/mistral/engine1/base.py.
5. Events should be published to a queue and then processed by a worker so
not to disrupt actual workflow/task executions.
6. API controller to register event subscriber
a. New resource type named EventSubscriber
b. New REST controller named EventSubscribersController and CRUD
operations
7. DB v2 sqlalchemy model named EventSubscriber and appropriate CRUD methods
8. Operations in python-mistralclient to manage CRUD for subscribers.
9. Operation in python-mistralclient to re-publish events for a given
workflow execution (in case where client applications was downed and need
the data to recover).
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Event Subscription

[W Chan]

Renat,

To clarify on the shortcut solution, are you saying 1) we add an adhoc
event subscription to the workflow spec OR 2) add a one time event
subscription to the workflow execution OR both?

I propose a separate worker/executor to process the events so to not
disrupt the workflow executions.  What if there are a lot of subscribers?
What if one or more subscribers are offline?  Do we retry and how many
times?  These activities will likely disrupt the throughput of the
workflows and I rather handle these activities separately.

Winson
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Event Subscription

[W Chan]

Renat,

Alternately, what do you think if mistral just post the events to given
exchange(s) on the same transport backend and let the subscribers decide
how to consume the events (i.e post to webhook, etc.) from these
exchanges?  This will simplify implementation somewhat.  The engine can
just take care of publishing the events to the exchanges and call it done.

Winson
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Event Subscription

[Renat Akhmerov]

Hi Winson,

> On 02 Dec 2014, at 09:10, W Chan  wrote:
> 
> To clarify on the shortcut solution, are you saying 1) we add an adhoc event 
> subscription to the workflow spec OR 2) add a one time event subscription to 
> the workflow execution OR both?

Not sure what you mean by “adhoc” here. What I meant is that we should have 2 
options:
Have an individual REST endpoint to be able to subscribe for certain types of 
events any time. For example, “Notifications about all workflow events for 
workflow name ‘my_workflow’” or “Notifications about switching to state ‘ERROR’ 
for workflow ‘my_workflow’”. Using this endpoint we can also unsubscribe from 
these events.
When we start a workflow (“mistral execution-create” in CLI and 
start_workflow() method in engine) we can configure the same subscription and 
pass it along with “start workflow” request. For such purposes, engine method 
“start_workflow” has keyword parameter “**params” that can take any kind of 
additional parameters needed for workflow start (for example, when we start a 
reverse workflow we pass “task_name”). This way we can start a workflow and 
configure our subscription with a single request. In the first approach we 
would have to make two requests.

> I propose a separate worker/executor to process the events so to not disrupt 
> the workflow executions.  What if there are a lot of subscribers?  What if 
> one or more subscribers are offline?  Do we retry and how many times?  These 
> activities will likely disrupt the throughput of the workflows and I rather 
> handle these activities separately.


Yeah, I now tend to agree with you here. Although it still bothers me because 
of that performance overhead that we’ll have most of the time. But generally, 
yes, you’re right. Do you think we should use same executors to process these 
notifications or introduce a new type of entity for that?

Thanks

Renat Akhmerov
@ Mirantis Inc.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Event Subscription

[W Chan]

Renat,

I agree with the two methods you proposed.

On processing the events, I was thinking a separate entity.  But you gave
me an idea, how about a system action for publishing the events that the
current executors can run?

Alternately, instead of making HTTP calls, what do you think if mistral
just post the events to the exchange(s) that the subscribers provided and
let the subscribers decide how to consume the events (i.e post to webhook,
etc.) from these exchanges?  This will simplify implementation somewhat.
The engine can just take care of publishing the events to the exchanges and
call it done.

Winson
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Event Subscription

[Renat Akhmerov]

> On 02 Dec 2014, at 23:53, W Chan  wrote:
> 
> On processing the events, I was thinking a separate entity.  But you gave me 
> an idea, how about a system action for publishing the events that the current 
> executors can run?

Yes, sounds great! I really like the idea.

> Alternately, instead of making HTTP calls, what do you think if mistral just 
> post the events to the exchange(s) that the subscribers provided and let the 
> subscribers decide how to consume the events (i.e post to webhook, etc.) from 
> these exchanges?  This will simplify implementation somewhat.  The engine can 
> just take care of publishing the events to the exchanges and call it done.

Yep. I understand the general idea but am still a little confused. Can you 
please share the details of what you mean by “exchange” and who is going to 
consume events? If, like previously said, it will be just sending actions to 
executors then it’s ok, I got it. And you can just use executor client from 
rpc.py to do the job. Or here you mean something different? Would be nice it we 
could present a communication diagram (engine -> event occurred -> … ).


Thanks___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Event Subscription

[W Chan]

Renat,

On sending events to an "exchange", I mean an exchange on the transport
(i.e. rabbitMQ exchange
https://www.rabbitmq.com/tutorials/amqp-concepts.html).  On implementation
we can probably explore the notification feature in oslo.messaging.  But on
second thought, this would limit the consumers to trusted subsystems or
services though.  If we want the event consumers to be any 3rd party,
including untrusted, then maybe we should keep it as HTTP calls.

Winson
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Event Subscription

[Renat Akhmerov]

Ok, got it.

So my general suggestion here is: let's keep it as simple as possible for now, 
create something that works and then let’s see how to improve it. And yes, 
consumers may be and mostly will be 3rd parties.

Thanks

Renat Akhmerov
@ Mirantis Inc.



> On 09 Dec 2014, at 08:25, W Chan  wrote:
> 
> Renat,
> 
> On sending events to an "exchange", I mean an exchange on the transport (i.e. 
> rabbitMQ exchange https://www.rabbitmq.com/tutorials/amqp-concepts.html 
> ).  On implementation 
> we can probably explore the notification feature in oslo.messaging.  But on 
> second thought, this would limit the consumers to trusted subsystems or 
> services though.  If we want the event consumers to be any 3rd party, 
> including untrusted, then maybe we should keep it as HTTP calls.
> 
> Winson
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Event Subscription

[Nikolay Makhotkin]

Hi, Winson,

This is great!

But I have a question:

> 9. Operation in python-mistralclient to re-publish events for a given
> workflow execution (in case where client applications was downed and need
> the data to recover).

How does Mistral know about all the events which it has already sent to
subscriber?

I guess you propose to store this data in EventSubscriber object. Correct
me if I mistaked.

On Tue, Nov 11, 2014 at 11:28 PM, W Chan  wrote:

> Regarding blueprint to register event listeners to notify client
> applications on state changes (
> https://blueprints.launchpad.net/mistral/+spec/mistral-event-listeners-http),
> I want to propose the following.
>
> 1. Refer this feature as event subscription instead of callback
> 2. Event subscription supports only HTTP web hooks with retry policy in
> this first attempt
> 3. Event subscription can be defined for a list of specific events,
> workflows, projects, domains, or any combinations (all if list is empty).
> 4. Decorator to publish event (similar to logging) and place decorators at
> Engine and TaskPolicy methods @
> https://github.com/stackforge/mistral/blob/master/mistral/engine1/base.py
> .
> 5. Events should be published to a queue and then processed by a worker so
> not to disrupt actual workflow/task executions.
> 6. API controller to register event subscriber
> a. New resource type named EventSubscriber
> b. New REST controller named EventSubscribersController and CRUD
> operations
> 7. DB v2 sqlalchemy model named EventSubscriber and appropriate CRUD
> methods
> 8. Operations in python-mistralclient to manage CRUD for subscribers.
> 9. Operation in python-mistralclient to re-publish events for a given
> workflow execution (in case where client applications was downed and need
> the data to recover).
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Best Regards,
Nikolay
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Event Subscription

[W Chan]

Nikolay,

You're right.  We will need to store the events in order to re-publish.
How about a separate Event model?  The events are written to the DB by the
same worker that publishes the event.  The retention policy for these
events is then managed by a config option.

Winson
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Event Subscription

[Renat Akhmerov]

Ok.

Here’s what I think.

Mostly I like the solution you’re proposing. Some comments/questions:
We discussed this with Dmitry about a couple of months ago and we concluded 
that we don’t necessarily need a separate endpoint, CRUD operations etc. (even 
though my initial BP had exactly this idea). One of the reasons was that we can 
just attach required information about events we’re interested in to “start 
workflow” request (technically it may be that “params” argument in 
“start_workflow” method) so that you don’t need to make 2 requests: 1) 
Subscribe to required events 2) Start workflow. However, I think we could 
combine these two things: having a separate endpoint and being able to attach 
“callback” information (the naming should be different though) in “start 
workflow” request. The first option is useful when I’m not interested in some 
particular workflow. For example, if I as a Mistral client just want to monitor 
what’s happening with my multiple workflows (not certain executions but rather 
all executions of specific workflows) and want to react on them somehow. The 
second one is just a shortcut solution to avoid one extra request to make event 
subscription.
Having only decorators is not enough. Even though I like that idea indeed. For 
instance, the event of completing workflow can’t be handled by a decorator I 
guess. But I’m not 100% sure, it may actually depend on how smart that 
decorator is. Anyway, just a thought.
Do you think we need a separate worker/executor connected via MQ to make all 
the notifications? Would it be simpler to just make those HTTP calls directly 
from engine w/o distributing them? Seems like distribution overhead may be 
higher than making the call itself. Btw, we’re now thinking about the concept 
of ‘local actions’ that don’t have to be distributed across executors. The 
reason is the same: distribution overhead is much higher than just running them 
right in the engine process. So I would like to know what you think here.

Thanks

Renat Akhmerov
@ Mirantis Inc.



> On 12 Nov 2014, at 22:21, W Chan  wrote:
> 
> Nikolay,
> 
> You're right.  We will need to store the events in order to re-publish.  How 
> about a separate Event model?  The events are written to the DB by the same 
> worker that publishes the event.  The retention policy for these events is 
> then managed by a config option.
> 
> Winson
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev