Hi Michael,

Interesting analysis.

We have had some discussions regarding the prediction/ regression
improvement in dev list.

It was based around introducing curve fitting, rather than using separate "ave
+ ave grad + ave 2nd grad", what we have now.

The discussion was under the subject "[Autoscaling] [Improvement]
Introducing "curve fitting" for stat prediction algorithm of Autoscaler".

Could you have a look at that discussion as well. Also i have filled the
same suggestion in a Jira [1].

Thanks.

[1] https://issues.apache.org/jira/browse/STRATOS-1211

On Fri, Feb 13, 2015 at 9:37 PM, Michael Hall (michaha2) <micha...@cisco.com
> wrote:

>  Hi Imesh, Lakmal, Devs,
>
>  Following the attached email thread, this email is intended as a
> starting point in formalising the proposal of the proceeding autoscale
> enhancement.
>
>  Kind regards,
>
>  Mike…
>
>  Proposal for ‘*Transition Compensated Continuous Scaling*’ enhancement
> to be added to Apache Stratos Autoscale feature to:
>
>    1. Greatly improve (~ x100) the maximum rate of cluster size increase
>    (maximum rate of ascent), when subjected to a sudden increase in load.
>    (continuous scaling decisions can occur as the decision isn’t delayed
>    (cluster monitor interval) to wait for the system to tend toward a steady
>    state)
>    2.  Eliminate redundant cartridges being spawned/terminated because of
>    cartridge startup/stop being larger than a scaling decision interval
>    (cluster monitor interval)
>
> *Implementation Overview:*
>
>  Current:
>
>  Measured health statistic -> sent to CEP -> 1 minute average -> forward
> prediction (use ave + ave grad + ave 2nd grad) -> use autoscale policy to
> calc number of required cartridges -> compare required cartridge count, to
> current cartridge found and scale appropriately
>
>  Proposed:
>
>  Measured health statistic -> sent to CEP -> 1 minute ‘moving’ average (per
> second) -> forward prediction (use ave + ave grad + ave 2nd grad) -> use
> autoscale policy to calc number of required cartridges -> compare required
> cartridge count, to ( the active (current) cartridge count + the spawning
> cartridge count – the terminating cartridge count ) and scale
> appropriately
>
>  *Implementation of ‘spawning’/‘terminating’ cartridge count:*
>
>  Currently the autoscale feature is not aware of the amount of cartridges
> in the cluster that are transitioning to and from the ACTIVE state. The
> proposed enhancement relies on being able to know this count at any given
> moment in time.
>
>  This can be implemented by using asynchronous events, where:
>
>  ‘MEMBER SPAWNED EVENT’ -> increments cluster-cartridge-count-spawned
> ‘MEMBER ACTIVE EVENT’ -> decrements cluster-cartridge-count-spawned, and
> increments cluster-cartridge-count-active
> ‘MEMBER TERMINATING EVENT’ -> increments cluster-cartridge-count-
> terminating
> ‘MEMBER TERMINATED EVENT’ -> decrements cluster-cartridge-count-terminating,
> and decrements cluster-cartridge-count-active
>
>  *Summary*
>
>  By compensating the ‘current’ cartridge count/ cluster size, with the
> cartridges that are transitioning, we remove the issue of duplicating
> scaling decisions whilst also allowing the scaling decision to occur
> continuously, greatly improving our ‘maximum rate of ascent’ when scaling
> up our cluster in reaction to a sudden increase in load.
>
>
>
>   From: Michael Hall <micha...@cisco.com>
> Reply-To: "dev@stratos.apache.org" <dev@stratos.apache.org>
> Date: Friday, 13 February 2015 11:36
> To: Lakmal Warusawithana <lak...@wso2.com>, "dev@stratos.apache.org" <
> dev@stratos.apache.org>, Imesh Gunaratne <im...@wso2.com>
> Subject: Re: autoscale architecture
>
>   That’s a good plan,
>
>  My work number is +442088242650
>
>  I’m around now, but will break for a while for lunch in an hour or so.
>
>  Cheers
>
>   From: Lakmal Warusawithana <lak...@wso2.com>
> Date: Friday, 13 February 2015 11:24
> To: "dev@stratos.apache.org" <dev@stratos.apache.org>, Imesh Gunaratne <
> im...@wso2.com>, Michael Hall <micha...@cisco.com>
> Subject: Re: autoscale architecture
>
>   Shall we go for a call, it will be more productive.
>
> On Fri, Feb 13, 2015 at 4:45 PM, Lakmal Warusawithana <lak...@wso2.com>
> wrote:
>
>> Hi Michael
>>
>> On Fri, Feb 13, 2015 at 4:14 PM, Michael Hall (michaha2) <
>> micha...@cisco.com> wrote:
>>
>>>  Hi Imesh,
>>>
>>>  So ‘transistion compensated’ refers to cartridges, which are
>>> ’transistioning’ between SPAWNED-ACTIVE, and TERMINATING-TERMINATED.
>>>
>>>  What it really means, is that if the 'aggregated average’ (Referred to
>>> this as <metric>PredictedValue in scaling.drl) is compensated:
>>>
>>>    1. As if the ‘spawning’ cartridges are providing resouce (although
>>>    they aren’t yet)
>>>    2. As if the ‘terminating’ cartridges have removed resource
>>>    (although they haven't yet)
>>>
>>> Such that the ‘transition compensated aggregated average', will be
>>> approximately what the actually aggregated average would be if those
>>> cartridges had become fully ‘active’ or ‘terminated’. This means the
>>> ‘transition compensated aggregated average’ is always in a sensible state
>>> to make a scaling decision.
>>>
>>>  This then allows us to make a scaling decision as often as we’d like
>>> (much smaller than 90 seconds, could even be every 1 second), because if
>>> you take the example the we’ve scaled up, the 'transition compensated
>>> aggregated average’ will instantly adjust to N/N+1 of it’s raw value
>>> (copied formula from previous email for reference below), so another
>>> scaling decision will only occur, if the underlying load (aggregated
>>> average) increases even further.
>>>
>>>  *transistion-compensated-agg-ave = agg-ave * ( cluster-size /
>>> cluster-size +  cluster-spawned-size - cluster–terminating-size )*
>>>
>>>
>>   I think this is good proposal, definitely it will help to calculate
>> more accurate agg-ave values. Since CEP has the topology information we can
>> easily calculate this.
>>
>>  AFAIK, auto scaler take care of cartridge states when calculating
>> required instances count for a predicted load.
>>
>>
>>
>>>   I’d be more than happy to setup a webex meeting to try and explain
>>> this better? Or another avenue of communication at your preference?
>>>
>>>  Kind regards,
>>>
>>>  Mike
>>>
>>>   From: Imesh Gunaratne <im...@apache.org>
>>> Reply-To: "dev@stratos.apache.org" <dev@stratos.apache.org>
>>> Date: Friday, 13 February 2015 01:09
>>>
>>> To: dev <dev@stratos.apache.org>
>>> Subject: Re: autoscale architecture
>>>
>>>   Hi Mike,
>>>
>>>  Thanks for the detailed explanation of your question. Currently we do
>>> not have the capability to do this in runtime for a specific cartridge.
>>> However we could reduce the global scaling decision interval. This needs to
>>> be configured at three locations:
>>>
>>>  1. Cartridge agent statistics publishing interval (default: 15 seconds)
>>> 2. CEP execution plan/faulty member detection interval (default: 1 min)
>>> 3. Autoscaler cluster monitor interval (default: 90 seconds)
>>>
>>>  I did not clearly get what you mean by 'transition compensated'. Is
>>> there a way to explain it further?
>>>
>>>  Thanks
>>>
>>>
>>> On Fri, Feb 13, 2015 at 12:26 AM, Michael Hall (michaha2) <
>>> micha...@cisco.com> wrote:
>>>
>>>>  Hi Dev,
>>>>
>>>>  Thanks for your response Imesh, if its ok, I’d like to skip straight
>>>> to my (rather lengthy) question:
>>>>
>>>>  Does the autoscaler have, currently or plans to introduce, a means to
>>>> receive an asynchronous event, signalling that a cartridge has gone from
>>>> ‘SPAWNED’ to ‘ACTIVE’, after it is launched from a 'scale-up’ decision, so
>>>> that, scaling decision interval can decrease to approximately the metric
>>>> update interval, and multiple cartridges are not spawned when only one is
>>>> needed?
>>>>
>>>>  In more depth:
>>>>
>>>>  The reasons for my question being that by knowing a cartridge is in
>>>> the ‘SPAWNED’ or ’TERMINATING’ state, the aggregated metric averages can be
>>>> ’transition compensated’ I.e…
>>>> *transistion-compensated-agg-ave = agg-ave * ( cluster-size /
>>>> cluster-size +  cluster-spawned-size - cluster–terminating-size )*
>>>> To allow the scaling decisions to occur on a continuous (only throttled
>>>> by the metric update frequency) basis.
>>>>
>>>>  It appears that currently scaling decision occurs ~minutes. If this
>>>> becomes ~seconds, it would vastly improving the maximum rate of ascent a
>>>> cluster can scale against sudden increase in load.
>>>>
>>>>  It appears that there is no spawning state awareness, which also
>>>> means several ‘redundant’ instances get spawned, when instance startup time
>>>> is greater than the scale decision interval.
>>>>
>>>>  Finally:
>>>>
>>>>  Are there difficulties in tracking ‘SPAWNED’ to ‘ACTIVE’ state on a
>>>> per cartridge basis, how does this align (if its a valid enhancement) with
>>>> other potential improvements that could be made to the autoscaler?
>>>>
>>>>  Regards,
>>>>
>>>>  Mike
>>>>
>>>>   From: Imesh Gunaratne <im...@apache.org>
>>>> Reply-To: "dev@stratos.apache.org" <dev@stratos.apache.org>
>>>> Date: Thursday, 12 February 2015 18:16
>>>> To: dev <dev@stratos.apache.org>
>>>> Subject: Re: autoscale architecture
>>>>
>>>>   Hi Michael,
>>>>
>>>>  Yes you can ask any questions you have on Autoscaling here.
>>>>
>>>>  I don't think we have documented Autoscaling feature in 4.1.0 at the
>>>> moment. However you could find some information here [1]. Autoscaling has
>>>> slightly changed with Composite Application Model.
>>>>
>>>>  [1]
>>>> https://cwiki.apache.org/confluence/display/STRATOS/4.1.0+Autoscaler
>>>>
>>>>  Thanks
>>>>
>>>> On Thu, Feb 12, 2015 at 9:33 PM, Michael Hall (michaha2) <
>>>> micha...@cisco.com> wrote:
>>>>
>>>>>  Hi Devs,
>>>>>
>>>>>  Is there a resource or contact that can help me understand the
>>>>> current, and planned architecture of the autoscaling feature within 
>>>>> Stratos.
>>>>>
>>>>>  Best Regards,
>>>>>
>>>>>  Mike
>>>>>
>>>>
>>>>
>>>>
>>>>  --
>>>>  Imesh Gunaratne
>>>>
>>>> Technical Lead, WSO2
>>>> Committer & PMC Member, Apache Stratos
>>>>
>>>
>>>
>>>
>>>  --
>>>  Imesh Gunaratne
>>>
>>> Technical Lead, WSO2
>>> Committer & PMC Member, Apache Stratos
>>>
>>
>>
>>
>>  --
>>  Lakmal Warusawithana
>> Vice President, Apache Stratos
>> Director - Cloud Architecture; WSO2 Inc.
>> Mobile : +94714289692
>> Blog : http://lakmalsview.blogspot.com/
>>
>>
>
>
>  --
>  Lakmal Warusawithana
> Vice President, Apache Stratos
> Director - Cloud Architecture; WSO2 Inc.
> Mobile : +94714289692
> Blog : http://lakmalsview.blogspot.com/
>
>


-- 
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware

email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146

Reply via email to