HI Martin, Yes, we need to define a deployment policy for groups scaling enabled groups.
Good to hear that you have solved the issue. Thanks. On Thu, May 28, 2015 at 7:07 AM, Martin Eppel (meppel) <mep...@cisco.com> wrote: > Hi Lakmal, Udara, > > > > I think I found the issue – apparently for group scaling to work a > deployment-policy has to be defined at group level (can you confirm this, > are there special requirements for the group level deployment policies ?). > > > > Once I added it, groups started to scale and create respective VM > instances. I am still analyzing the logs but I think I am good for the > moment, > > > > Thanks > > > > Martin > > > > *From:* Martin Eppel (meppel) > *Sent:* Wednesday, May 27, 2015 9:06 AM > *To:* dev@stratos.apache.org; Udara Liyanage > > *Subject:* RE: Testing stratos 4.1: group scaling issue > > > > Thanks Lakmal, Udara > > > > *From:* Lakmal Warusawithana [mailto:lak...@wso2.com <lak...@wso2.com>] > *Sent:* Tuesday, May 26, 2015 11:00 PM > *To:* dev@stratos.apache.org; Udara Liyanage > *Subject:* Re: Testing stratos 4.1: group scaling issue > > > > Hi Udara, > > > > if you have some cycles, can you work with Martin on this issue. > > > > On Wed, May 27, 2015 at 6:02 AM, Martin Eppel (meppel) <mep...@cisco.com> > wrote: > > Attaching the proper deployment-policy-1.json, please disregard the one in > the zip file > > > > Thanks > > > > Martin > > > > *From:* Martin Eppel (meppel) > *Sent:* Tuesday, May 26, 2015 5:04 PM > *To:* dev@stratos.apache.org > *Subject:* RE: Testing stratos 4.1: group scaling issue > > > > > > Hi Reka, > > > > I tried various way to get the group scaling to work, so far without > success. > > The latest application I created is as shown below [1.] > > > > This is what I observe: > > > > The application starts up normally, with all (initial) cartridges going > active in the proper sequence > > > > From the logs: > > The very first time a scaling decision is made to scale up a group it > seems form the log that the max number is already reached: > > “ > > TID: [0] [STRATOS] [2015-05-26 02:07:51,628] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scale-up] Trying to scale > up over max, hence not scaling up cluster itself and > > notifying to parent for possible group scaling or > app bursting. > > [cluster] g-sc-G12-1.c5-2x0.c5.domain [instance > id]g-sc-G3-1-1x0-1 [max] 1 > > ” > > > > It is followed by a subsequent log entry complaining that the max limit > of 4 has been reached. I am assuming this refers back to the > “groupMaxInstances” in the application.json of group with alias "g-sc-G3-1". > > > > However, I don’t see any entries which indicate that group instances were > successfully created before hitting the max number. > > > > “ > > TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling max out > notification is going to the [parentInstance] g-sc-G3-1-1x0-1 > > TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG > {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} - > Child Scaling max out event received to [group]: g-sc-G3-1-1x0, [network > partition]: RegionOne, [event] g-sc-G12-1.c5-2x0.c5.domain, [group > instance] g-sc-G3-1-1x0-1 > > TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG > {org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor} - Rule > executed for: NetworkPartitionContext > [id=g-sc-G3-1-1x0-1partitionAlgorithm=one-after-another, > minInstanceCount=1, maxInstanceCount=1] > > TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG > {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group > monitor is running: [group] g-sc-G3-1-1x0 > > TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG > {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Handling > group scaling for the [group] g-sc-G3-1-1x0upon a max out event from the > children > > TID: [0] [STRATOS] [2015-05-26 02:07:51,629] WARN > {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - [Group] > g-sc-G3-1-1x0 has reached the maximum limit as [max] 4. Hence trying to > notify the parent. > > TID: [0] [STRATOS] [2015-05-26 02:07:51,630] DEBUG > {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} - > Child Scaling max out event received to [group]: g-sc-G2-1-0x0, [network > partition]: RegionOne, [event] g-sc-G12-1, [group instance] g-sc-G2-1-0x0-1 > > “ > > > > I checked the code and added some additional traces and it turns out that > the following code is executed: > > > > In *GroupMonitor.java-> createInstanceOnDemand()*, with *partitionContext > as being null* > > …. > > *if (partitionContext != null) {* > > * groupInstanceId = > createGroupInstanceAndAddToMonitor(group, parentInstanceContext,* > > * partitionContext,* > > * parentLevelNetworkPartitionContext,* > > * null);* > > * instanceIdsToStart.add(groupInstanceId);* > > * } else {* > > * log.warn("[Group] " + group.getUniqueIdentifier() + " > has reached " +* > > * "the maximum limit as [max] " + groupMax +* > > * ". Hence trying to notify the parent.");* > > * }* > > > > > > Also, no new cartridge instances were created beyond the initial ones. > > > > The same “patterns” seems to hold for all other groups within the > application, hence no new group instance seems to be created (beyond the > initial ones). > > > > I am not sure if I am hitting a configuration issue or an issue with group > scaling. > > Please see logs (wso2carbon-8.log debug enabled and artifacts (polices, > application.json, cartridge-group.json)) in the attached zip file. > > > > At this point I don’t think this is a configuration issue but rather a > system issue (please advise if otherwise) , and I will open a JIRA > > > > Thanks > > > > Martin > > > > > > [1.] application > > > > > > > > > > > > > > > > *From:* Martin Eppel (meppel) > *Sent:* Friday, May 22, 2015 4:42 PM > *To:* dev@stratos.apache.org > *Subject:* RE: Testing stratos 4.1: group scaling > > > > Hi Reka, > > > > I created the following application – see [1.]. > > > > I am able to force the autoscaler to decide to scale up but something in > my configuration doesn’t seem to be right since it seems to have reached > the maximum of instances. I attached the autoscaling policies, deployment > policies, as well as cartridge-group.json, application.json and > wso2carbon.log to the email. Any pointers would be highly appreciated to > what’s missing or incorrectly configured > > > > Thanks > > > > Martin > > > > > > TID: [0] [STRATOS] [2015-05-22 23:03:14,865] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scale-up] Trying to scale > up over max, hence not scaling up cluster itself and > > notifying to parent for possible group scaling or > app bursting. > > [cluster] g-sc-G12-1.c3-1x0.c3.domain [instance > id]g-sc-G2-1-0x0-1 [max] 1 > > > > > > > > > > [1.] application > > > > > > > > > > *From:* Reka Thirunavukkarasu [mailto:r...@wso2.com <r...@wso2.com>] > *Sent:* Friday, May 22, 2015 10:41 AM > *To:* dev > *Subject:* Re: Testing stratos 4.1: group scaling > > > > Hi Martin, > > If you set the threshold of load average in the autoscaling policy to like > 20-40 range and use the below script[1] to stress a particular VM/Vms, then > the CPU usage will get higher. In that case, it will eventually exceed the > threshold and then stratos will try to scaleup. > > If you are using memory consumption, then you can try to increase the > memory usage using any of your scenarios and make the stratos to scaleup. > > However, if you set the threshold as very lower values, then even at the > beginning stratos will scaleup as the stats will hit the threshold. > > [1] > https://github.com/lahirus/SetupStratos/blob/master/stress/stress_yes.sh > > Thanks, > > Reka > > > > On Fri, May 22, 2015 at 10:01 PM, Martin Eppel (meppel) <mep...@cisco.com> > wrote: > > Hi Reka, > > > > Thanks for the explanation. I think my question was more directed towards > how can I simulate the statistics to exceed the thresholds to trigger the > scale up, especially in the context of test automation. > > > > So to rephrase it, how can I simulate cartridge statistics events without > having to modify the cartridge code ? > > > > Thanks > > > > Martin > > > > *From:* Reka Thirunavukkarasu [mailto:r...@wso2.com] > *Sent:* Thursday, May 21, 2015 9:59 PM > *To:* dev > *Subject:* Re: Testing stratos 4.1: group scaling > > > > Hi Martin, > > Let me explain how group scaling works in a high level manner. Say, it a > group(G1) has two independent clusters(no dependent scaling) C1 and C2. > > G1 - min 1 max5 > > C1 - min 1 max2 (applies per cluster instance) > > C2 - min 2 max 3 (applies per cluster instance) > > Where C1 and C2 are children of G1. At the beginning, we will create one > group instance say G11 and corresponding cluster instance for C1 and C2 of > G11 will get created. Then each C1 and C2's cluster instance will receive > stats per cluster instance wise. If the stats is greater than the > threshold, then as C1 and C2's cluster instance have room to scaleup by one > more instance according min and max configuration, each clusters will get > scaleup until it hits the max. Once max is reached, if the stats is still > high, then when C1 or C2 doesn't have space to scaleup, it will notify the > parent (G1). So, G1 will check whether it has group scaling capability. If > so, it will try to create new group instance. If no group scaling enable, > it will notify its parent. If the eventual parent application, then > application will try to get busted if there is space. > > If you mention your configuration as below, then when the stats is higher, > group scaling will immediately happen as C1 and C2 don't have space to > scaleup: > > G1 - min 1 max5 > C1 - min 1 max1 > C2 - min 2 max 2 > > Hope above points will help you to test the group scaling. > > > > Thanks, > > Reka > > > > > > On Fri, May 22, 2015 at 5:20 AM, Martin Eppel (meppel) <mep...@cisco.com> > wrote: > > Hi, > > > > I am setting up a test application to test group scaling (following the > example from the 4.1 Wiki - > https://cwiki.apache.org/confluence/display/STRATOS/4.1.0+Group+Scaling+Application+on+OpenStack). > > > > > I was wondering if you have any suggestion how to actually force / > simulate the conditions to force a group to scale up ? How does it work in > the sample application (to get the group to scale up) ? > > > > Btw I am using open stack as IaaS, not the Mock IaaS. > > > > Thanks > > > > Martin > > > > > > > -- > > Reka Thirunavukkarasu > Senior Software Engineer, > WSO2, Inc.:http://wso2.com, > > Mobile: +94776442007 > > > > > > > -- > > Reka Thirunavukkarasu > Senior Software Engineer, > WSO2, Inc.:http://wso2.com, > > Mobile: +94776442007 > > > > > > > > -- > > Lakmal Warusawithana > > Vice President, Apache Stratos > > Director - Cloud Architecture; WSO2 Inc. > > Mobile : +94714289692 > > Blog : http://lakmalsview.blogspot.com/ > -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware phone: +94773325954 email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
