Hi Udara,
if you have some cycles, can you work with Martin on this issue.
On Wed, May 27, 2015 at 6:02 AM, Martin Eppel (meppel) <mep...@cisco.com>
wrote:
> Attaching the proper deployment-policy-1.json, please disregard the one
> in the zip file
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Tuesday, May 26, 2015 5:04 PM
> *To:* dev@stratos.apache.org
> *Subject:* RE: Testing stratos 4.1: group scaling issue
>
>
>
>
>
> Hi Reka,
>
>
>
> I tried various way to get the group scaling to work, so far without
> success.
>
> The latest application I created is as shown below [1.]
>
>
>
> This is what I observe:
>
>
>
> The application starts up normally, with all (initial) cartridges going
> active in the proper sequence
>
>
>
> From the logs:
>
> The very first time a scaling decision is made to scale up a group it
> seems form the log that the max number is already reached:
>
> “
>
> TID: [0] [STRATOS] [2015-05-26 02:07:51,628] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scale-up] Trying to scale
> up over max, hence not scaling up cluster itself and
>
> notifying to parent for possible group scaling or
> app bursting.
>
> [cluster] g-sc-G12-1.c5-2x0.c5.domain [instance
> id]g-sc-G3-1-1x0-1 [max] 1
>
> ”
>
>
>
> It is followed by a subsequent log entry complaining that the max limit
> of 4 has been reached. I am assuming this refers back to the
> “groupMaxInstances” in the application.json of group with alias "g-sc-G3-1".
>
>
>
> However, I don’t see any entries which indicate that group instances were
> successfully created before hitting the max number.
>
>
>
> “
>
> TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling max out
> notification is going to the [parentInstance] g-sc-G3-1-1x0-1
>
> TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG
> {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} -
> Child Scaling max out event received to [group]: g-sc-G3-1-1x0, [network
> partition]: RegionOne, [event] g-sc-G12-1.c5-2x0.c5.domain, [group
> instance] g-sc-G3-1-1x0-1
>
> TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG
> {org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor} - Rule
> executed for: NetworkPartitionContext
> [id=g-sc-G3-1-1x0-1partitionAlgorithm=one-after-another,
> minInstanceCount=1, maxInstanceCount=1]
>
> TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG
> {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group
> monitor is running: [group] g-sc-G3-1-1x0
>
> TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG
> {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Handling
> group scaling for the [group] g-sc-G3-1-1x0upon a max out event from the
> children
>
> TID: [0] [STRATOS] [2015-05-26 02:07:51,629] WARN
> {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - [Group]
> g-sc-G3-1-1x0 has reached the maximum limit as [max] 4. Hence trying to
> notify the parent.
>
> TID: [0] [STRATOS] [2015-05-26 02:07:51,630] DEBUG
> {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} -
> Child Scaling max out event received to [group]: g-sc-G2-1-0x0, [network
> partition]: RegionOne, [event] g-sc-G12-1, [group instance] g-sc-G2-1-0x0-1
>
> “
>
>
>
> I checked the code and added some additional traces and it turns out that
> the following code is executed:
>
>
>
> In *GroupMonitor.java-> createInstanceOnDemand()*, with *partitionContext
> as being null*
>
> ….
>
> *if (partitionContext != null) {*
>
> * groupInstanceId =
> createGroupInstanceAndAddToMonitor(group, parentInstanceContext,*
>
> * partitionContext,*
>
> * parentLevelNetworkPartitionContext,*
>
> * null);*
>
> * instanceIdsToStart.add(groupInstanceId);*
>
> * } else {*
>
> * log.warn("[Group] " + group.getUniqueIdentifier() + "
> has reached " +*
>
> * "the maximum limit as [max] " + groupMax +*
>
> * ". Hence trying to notify the parent.");*
>
> * }*
>
>
>
>
>
> Also, no new cartridge instances were created beyond the initial ones.
>
>
>
> The same “patterns” seems to hold for all other groups within the
> application, hence no new group instance seems to be created (beyond the
> initial ones).
>
>
>
> I am not sure if I am hitting a configuration issue or an issue with group
> scaling.
>
> Please see logs (wso2carbon-8.log debug enabled and artifacts (polices,
> application.json, cartridge-group.json)) in the attached zip file.
>
>
>
> At this point I don’t think this is a configuration issue but rather a
> system issue (please advise if otherwise) , and I will open a JIRA
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> [1.] application
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Friday, May 22, 2015 4:42 PM
> *To:* dev@stratos.apache.org
> *Subject:* RE: Testing stratos 4.1: group scaling
>
>
>
> Hi Reka,
>
>
>
> I created the following application – see [1.].
>
>
>
> I am able to force the autoscaler to decide to scale up but something in
> my configuration doesn’t seem to be right since it seems to have reached
> the maximum of instances. I attached the autoscaling policies, deployment
> policies, as well as cartridge-group.json, application.json and
> wso2carbon.log to the email. Any pointers would be highly appreciated to
> what’s missing or incorrectly configured
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> TID: [0] [STRATOS] [2015-05-22 23:03:14,865] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scale-up] Trying to scale
> up over max, hence not scaling up cluster itself and
>
> notifying to parent for possible group scaling or
> app bursting.
>
> [cluster] g-sc-G12-1.c3-1x0.c3.domain [instance
> id]g-sc-G2-1-0x0-1 [max] 1
>
>
>
>
>
>
>
>
>
> [1.] application
>
>
>
>
>
>
>
>
>
> *From:* Reka Thirunavukkarasu [mailto:r...@wso2.com <r...@wso2.com>]
> *Sent:* Friday, May 22, 2015 10:41 AM
> *To:* dev
> *Subject:* Re: Testing stratos 4.1: group scaling
>
>
>
> Hi Martin,
>
> If you set the threshold of load average in the autoscaling policy to like
> 20-40 range and use the below script[1] to stress a particular VM/Vms, then
> the CPU usage will get higher. In that case, it will eventually exceed the
> threshold and then stratos will try to scaleup.
>
> If you are using memory consumption, then you can try to increase the
> memory usage using any of your scenarios and make the stratos to scaleup.
>
> However, if you set the threshold as very lower values, then even at the
> beginning stratos will scaleup as the stats will hit the threshold.
>
> [1]
> https://github.com/lahirus/SetupStratos/blob/master/stress/stress_yes.sh
>
> Thanks,
>
> Reka
>
>
>
> On Fri, May 22, 2015 at 10:01 PM, Martin Eppel (meppel) <mep...@cisco.com>
> wrote:
>
> Hi Reka,
>
>
>
> Thanks for the explanation. I think my question was more directed towards
> how can I simulate the statistics to exceed the thresholds to trigger the
> scale up, especially in the context of test automation.
>
>
>
> So to rephrase it, how can I simulate cartridge statistics events without
> having to modify the cartridge code ?
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Reka Thirunavukkarasu [mailto:r...@wso2.com]
> *Sent:* Thursday, May 21, 2015 9:59 PM
> *To:* dev
> *Subject:* Re: Testing stratos 4.1: group scaling
>
>
>
> Hi Martin,
>
> Let me explain how group scaling works in a high level manner. Say, it a
> group(G1) has two independent clusters(no dependent scaling) C1 and C2.
>
> G1 - min 1 max5
>
> C1 - min 1 max2 (applies per cluster instance)
>
> C2 - min 2 max 3 (applies per cluster instance)
>
> Where C1 and C2 are children of G1. At the beginning, we will create one
> group instance say G11 and corresponding cluster instance for C1 and C2 of
> G11 will get created. Then each C1 and C2's cluster instance will receive
> stats per cluster instance wise. If the stats is greater than the
> threshold, then as C1 and C2's cluster instance have room to scaleup by one
> more instance according min and max configuration, each clusters will get
> scaleup until it hits the max. Once max is reached, if the stats is still
> high, then when C1 or C2 doesn't have space to scaleup, it will notify the
> parent (G1). So, G1 will check whether it has group scaling capability. If
> so, it will try to create new group instance. If no group scaling enable,
> it will notify its parent. If the eventual parent application, then
> application will try to get busted if there is space.
>
> If you mention your configuration as below, then when the stats is higher,
> group scaling will immediately happen as C1 and C2 don't have space to
> scaleup:
>
> G1 - min 1 max5
> C1 - min 1 max1
> C2 - min 2 max 2
>
> Hope above points will help you to test the group scaling.
>
>
>
> Thanks,
>
> Reka
>
>
>
>
>
> On Fri, May 22, 2015 at 5:20 AM, Martin Eppel (meppel) <mep...@cisco.com>
> wrote:
>
> Hi,
>
>
>
> I am setting up a test application to test group scaling (following the
> example from the 4.1 Wiki -
> https://cwiki.apache.org/confluence/display/STRATOS/4.1.0+Group+Scaling+Application+on+OpenStack).
>
>
>
>
> I was wondering if you have any suggestion how to actually force /
> simulate the conditions to force a group to scale up ? How does it work in
> the sample application (to get the group to scale up) ?
>
>
>
> Btw I am using open stack as IaaS, not the Mock IaaS.
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>
--
Lakmal Warusawithana
Vice President, Apache Stratos
Director - Cloud Architecture; WSO2 Inc.
Mobile : +94714289692
Blog : http://lakmalsview.blogspot.com/
