Re: [openstack-dev] [heat] One more lifecycle plug point - in scaling groups

[Zane Bitter]

On 01/07/14 16:30, Mike Spreitzer wrote:

Thinking about my favorite use case for lifecycle plug points for cloud
providers (i.e., giving something a chance to make a holistic placement
decision), it occurs to me that one more is needed: a scale-down plug
point.  A plugin for this point has a distinctive job: to decide which
group member(s) to remove from a scaling group (i.e.,
OS::Heat::AutoScalingGroup or OS::Heat::InstanceGroup or
OS::Heat::ResourceGroup or AWS::AutoScaling::AutoScalingGroup).  The
plugin's signature could be something like this: given a list of group
members and a number to remove, return the list of members to remove
(or, equivalently, return the list of members to keep).  What do you think?


I think you're not thinking big enough ;)

There exist a whole class of applications that would benefit from 
autoscaling but which are not quite stateless. (For example, a PaaS.) So 
it's not enough to have plugins that place the choice of which node to 
scale down under operator control; in fact it needs to be under 
_application_ control.


This is on the roadmap, and TripleO really needs it, so hopefully it 
will happen in Juno.


cheers,
Zane.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] One more lifecycle plug point - in scaling groups

[Mike Spreitzer]

Zane Bitter  wrote on 07/01/2014 07:05:15 PM:

> On 01/07/14 16:30, Mike Spreitzer wrote:
> > Thinking about my favorite use case for lifecycle plug points for 
cloud
> > providers (i.e., giving something a chance to make a holistic 
placement
> > decision), it occurs to me that one more is needed: a scale-down plug
> > point.  A plugin for this point has a distinctive job: to decide which
> > group member(s) to remove from a scaling group (i.e.,
> > OS::Heat::AutoScalingGroup or OS::Heat::InstanceGroup or
> > OS::Heat::ResourceGroup or AWS::AutoScaling::AutoScalingGroup).  The
> > plugin's signature could be something like this: given a list of group
> > members and a number to remove, return the list of members to remove
> > (or, equivalently, return the list of members to keep).  What do you 
think?
> 
> I think you're not thinking big enough ;)

I agree, I was taking only a small bite in hopes of a quick success.

> There exist a whole class of applications that would benefit from 
> autoscaling but which are not quite stateless. (For example, a PaaS.) So 

> it's not enough to have plugins that place the choice of which node to 
> scale down under operator control; in fact it needs to be under 
> _application_ control.

Exactly.  There are two different roles that want such control; in 
general, neither is happy if only the other gets it.  Now the question 
becomes, how do we get them to play nice together?  In the case of TripleO 
there may be an exceptionally easy out: the owner of an application 
deployed on the undercloud may well be the same as the provider of the 
undercloud (i.e., the operator whose end goal is to provide the 
overcloud(s) ).

> This is on the roadmap, and TripleO really needs it, so hopefully it 
> will happen in Juno.

I assume you mean giving this control to the application, which I presume 
amounts to giving it to the template author.  Is this written up 
somewhere?

Thanks,
Mike
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] One more lifecycle plug point - in scaling groups

[Adrian Otto]

Zane,

If you happen to have a link to this blueprint, could you reply with it? I took 
a look, but did not find it.

I’d like to suggest that the implementation allow apps to call unauthenticated 
(signed) webhook URLs in order to trigger a scale up/down event within a 
scaling group. This is about the simplest possible API to allow any application 
to control it’s own elasticity.

Thanks,

Adrian

On Jul 1, 2014, at 6:09 PM, Mike Spreitzer 
mailto:mspre...@us.ibm.com>> wrote:

Zane Bitter mailto:zbit...@redhat.com>> wrote on 07/01/2014 
07:05:15 PM:

> On 01/07/14 16:30, Mike Spreitzer wrote:
> > Thinking about my favorite use case for lifecycle plug points for cloud
> > providers (i.e., giving something a chance to make a holistic placement
> > decision), it occurs to me that one more is needed: a scale-down plug
> > point.  A plugin for this point has a distinctive job: to decide which
> > group member(s) to remove from a scaling group (i.e.,
> > OS::Heat::AutoScalingGroup or OS::Heat::InstanceGroup or
> > OS::Heat::ResourceGroup or AWS::AutoScaling::AutoScalingGroup).  The
> > plugin's signature could be something like this: given a list of group
> > members and a number to remove, return the list of members to remove
> > (or, equivalently, return the list of members to keep).  What do you think?
>
> I think you're not thinking big enough ;)

I agree, I was taking only a small bite in hopes of a quick success.

> There exist a whole class of applications that would benefit from
> autoscaling but which are not quite stateless. (For example, a PaaS.) So
> it's not enough to have plugins that place the choice of which node to
> scale down under operator control; in fact it needs to be under
> _application_ control.

Exactly.  There are two different roles that want such control; in general, 
neither is happy if only the other gets it.  Now the question becomes, how do 
we get them to play nice together?  In the case of TripleO there may be an 
exceptionally easy out: the owner of an application deployed on the undercloud 
may well be the same as the provider of the undercloud (i.e., the operator 
whose end goal is to provide the overcloud(s) ).

> This is on the roadmap, and TripleO really needs it, so hopefully it
> will happen in Juno.

I assume you mean giving this control to the application, which I presume 
amounts to giving it to the template author.  Is this written up somewhere?

Thanks,
Mike
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] One more lifecycle plug point - in scaling groups

[Steven Hardy]

On Wed, Jul 02, 2014 at 02:41:19AM +, Adrian Otto wrote:
>Zane,
>If you happen to have a link to this blueprint, could you reply with it? I
>took a look, but did not find it.
>I'd like to suggest that the implementation allow apps to call
>unauthenticated (signed) webhook URLs in order to trigger a scale up/down
>event within a scaling group. This is about the simplest possible API to
>allow any application to control it's own elasticity.

This is already possible, you just need to get the scaling policy alarm URL
into the instance where the application is running.

I believe Zane was referring to:

https://blueprints.launchpad.net/heat/+spec/update-hooks

This is also related to the action aware software config spec:

https://review.openstack.org/#/c/98742/

So in future, you might autoscale nested stacks containing action-aware
software config resources, then you could define specific actions which
happen e.g on scale-down (on action DELETE).

Steve

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] One more lifecycle plug point - in scaling groups

[Zane Bitter]

On 01/07/14 21:09, Mike Spreitzer wrote:

Zane Bitter  wrote on 07/01/2014 07:05:15 PM:

 > On 01/07/14 16:30, Mike Spreitzer wrote:
 > > Thinking about my favorite use case for lifecycle plug points for cloud
 > > providers (i.e., giving something a chance to make a holistic placement
 > > decision), it occurs to me that one more is needed: a scale-down plug
 > > point.  A plugin for this point has a distinctive job: to decide which
 > > group member(s) to remove from a scaling group (i.e.,
 > > OS::Heat::AutoScalingGroup or OS::Heat::InstanceGroup or
 > > OS::Heat::ResourceGroup or AWS::AutoScaling::AutoScalingGroup).  The
 > > plugin's signature could be something like this: given a list of group
 > > members and a number to remove, return the list of members to remove
 > > (or, equivalently, return the list of members to keep).  What do
you think?
 >
 > I think you're not thinking big enough ;)

I agree, I was taking only a small bite in hopes of a quick success.

 > There exist a whole class of applications that would benefit from
 > autoscaling but which are not quite stateless. (For example, a PaaS.) So
 > it's not enough to have plugins that place the choice of which node to
 > scale down under operator control; in fact it needs to be under
 > _application_ control.

Exactly.  There are two different roles that want such control; in
general, neither is happy if only the other gets it.  Now the question
becomes, how do we get them to play nice together?  In the case of
TripleO there may be an exceptionally easy out: the owner of an
application deployed on the undercloud may well be the same as the
provider of the undercloud (i.e., the operator whose end goal is to
provide the overcloud(s) ).


Let's assume that this feature takes that form of some additional data 
in the alarm trigger (i.e. the input to the scaling policy) that 
specifies which server(s) to delete first. The application would handle 
this by receiving the trigger from Ceilometer (or wherever) itself, and 
then inserting the additional data before passing it to Heat/autoscaling.


That gives us three options for e.g. a holistic scheduler to insert 
hints as to which servers to delete:


(a) Insert them into the outgoing triggers from Ceilometer. The 
application has the choice to override.


(b) Let the user explicitly configure the flow of notifications. So it 
could be any of:

Ceilometer -> Heat
Ceilometer -> Scheduler -> Heat
Ceilometer -> Application -> Heat
Ceilometer -> Scheduler -> Application -> Heat

(c) Insert them into incoming triggers in Heat whenever the application 
has not specified them. This is basically your original proposal.


I'm guessing that, of those, (c) is probably the winner. But we'd need 
to have that debate.


Another possible implementation is to do it with a notification and 
reply, rather than including it in the existing datapath.



 > This is on the roadmap, and TripleO really needs it, so hopefully it
 > will happen in Juno.

I assume you mean giving this control to the application, which I
presume amounts to giving it to the template author.  Is this written up
somewhere?


I had a quick look, but it doesn't appear we have a blueprint for it 
yet, unless you count the notifications blueprint that Steve mentioned 
(but I don't think that addresses this case specifically).


cheers,
Zane.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] One more lifecycle plug point - in scaling groups

[Mike Spreitzer]

Steven Hardy  wrote on 07/02/2014 06:16:21 AM:

> On Wed, Jul 02, 2014 at 02:41:19AM +, Adrian Otto wrote:
> >Zane,
> >If you happen to have a link to this blueprint, could you replywith 
it? ...
>
> I believe Zane was referring to:
> 
> https://blueprints.launchpad.net/heat/+spec/update-hooks
> 
> This is also related to the action aware software config spec:
> 
> https://review.openstack.org/#/c/98742/
> 
> So in future, you might autoscale nested stacks containing action-aware
> software config resources, then you could define specific actions which
> happen e.g on scale-down (on action DELETE).

Thanks, those are great pointers.  The second pretty much covers the 
first, right?

I do think the issue these address --- the need to get application logic 
involved in, e.g., shutdown --- is most of what an application needs; 
involvement in selection of which member(s) to delete is much less 
important (provided that clean shutdown mechanism prevents concurrent 
shutdowns).  So that provides a pretty good decoupling between the 
application's concerns and a holistic scheduler's concerns.

Thanks,
Mike

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] One more lifecycle plug point - in scaling groups

[Chris Friesen]

On 07/03/2014 10:13 PM, Mike Spreitzer wrote:


I do think the issue these address --- the need to get application logic
involved in, e.g., shutdown --- is most of what an application needs;
involvement in selection of which member(s) to delete is much less
important (provided that clean shutdown mechanism prevents concurrent
shutdowns).


I assume this is more of the whole "cattle" model, where an instance 
could disappear at any time so applications should design for that?


As an alternate viewpoint, if a particular instance in a group is 
working on something "expensive" (long-running, difficult to checkpoint, 
etc.), maybe it would make sense to allow the application to help make 
the decision on which instance to shut down (or possibly even veto/delay 
the scale down operation).  If it takes a minute to finish the 
operation, and 15 minutes to redo it on another instance...


Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev