Hi Martin, I just noticed that you have found the issue and fixed it. Glad to hear.
On Thu, May 28, 2015 at 10:15 AM, Udara Liyanage <ud...@wso2.com> wrote: > Hi Martin, > > I will try your scenario in Mock IaaS and let you know the results > > On Thu, May 28, 2015 at 7:49 AM, Lakmal Warusawithana <lak...@wso2.com> > wrote: > >> Hi Martin >> >> On Thu, May 28, 2015 at 7:07 AM, Martin Eppel (meppel) <mep...@cisco.com> >> wrote: >> >>> Hi Lakmal, Udara, >>> >>> >>> >>> I think I found the issue – apparently for group scaling to work a >>> deployment-policy has to be defined at group level (can you confirm this, >>> are there special requirements for the group level deployment policies ?). >>> >>> >>> >> >> Yes, same as individual cartridge scaling, we have to define where to >> scale up etc. Since we need group scaling, we need to define deployment >> policy in group level to say where to scale up the group. >> >> >>> Once I added it, groups started to scale and create respective VM >>> instances. I am still analyzing the logs but I think I am good for the >>> moment, >>> >>> >>> >>> Thanks >>> >>> >>> >>> Martin >>> >>> >>> >>> *From:* Martin Eppel (meppel) >>> *Sent:* Wednesday, May 27, 2015 9:06 AM >>> *To:* dev@stratos.apache.org; Udara Liyanage >>> >>> *Subject:* RE: Testing stratos 4.1: group scaling issue >>> >>> >>> >>> Thanks Lakmal, Udara >>> >>> >>> >>> *From:* Lakmal Warusawithana [mailto:lak...@wso2.com <lak...@wso2.com>] >>> *Sent:* Tuesday, May 26, 2015 11:00 PM >>> *To:* dev@stratos.apache.org; Udara Liyanage >>> *Subject:* Re: Testing stratos 4.1: group scaling issue >>> >>> >>> >>> Hi Udara, >>> >>> >>> >>> if you have some cycles, can you work with Martin on this issue. >>> >>> >>> >>> On Wed, May 27, 2015 at 6:02 AM, Martin Eppel (meppel) <mep...@cisco.com> >>> wrote: >>> >>> Attaching the proper deployment-policy-1.json, please disregard the one >>> in the zip file >>> >>> >>> >>> Thanks >>> >>> >>> >>> Martin >>> >>> >>> >>> *From:* Martin Eppel (meppel) >>> *Sent:* Tuesday, May 26, 2015 5:04 PM >>> *To:* dev@stratos.apache.org >>> *Subject:* RE: Testing stratos 4.1: group scaling issue >>> >>> >>> >>> >>> >>> Hi Reka, >>> >>> >>> >>> I tried various way to get the group scaling to work, so far without >>> success. >>> >>> The latest application I created is as shown below [1.] >>> >>> >>> >>> This is what I observe: >>> >>> >>> >>> The application starts up normally, with all (initial) cartridges going >>> active in the proper sequence >>> >>> >>> >>> From the logs: >>> >>> The very first time a scaling decision is made to scale up a group it >>> seems form the log that the max number is already reached: >>> >>> “ >>> >>> TID: [0] [STRATOS] [2015-05-26 02:07:51,628] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scale-up] Trying to scale >>> up over max, hence not scaling up cluster itself and >>> >>> notifying to parent for possible group scaling >>> or app bursting. >>> >>> [cluster] g-sc-G12-1.c5-2x0.c5.domain [instance >>> id]g-sc-G3-1-1x0-1 [max] 1 >>> >>> ” >>> >>> >>> >>> It is followed by a subsequent log entry complaining that the max limit >>> of 4 has been reached. I am assuming this refers back to the >>> “groupMaxInstances” in the application.json of group with alias "g-sc-G3-1". >>> >>> >>> >>> However, I don’t see any entries which indicate that group instances >>> were successfully created before hitting the max number. >>> >>> >>> >>> “ >>> >>> TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG >>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling max out >>> notification is going to the [parentInstance] g-sc-G3-1-1x0-1 >>> >>> TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG >>> {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} - >>> Child Scaling max out event received to [group]: g-sc-G3-1-1x0, [network >>> partition]: RegionOne, [event] g-sc-G12-1.c5-2x0.c5.domain, [group >>> instance] g-sc-G3-1-1x0-1 >>> >>> TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG >>> {org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor} - Rule >>> executed for: NetworkPartitionContext >>> [id=g-sc-G3-1-1x0-1partitionAlgorithm=one-after-another, >>> minInstanceCount=1, maxInstanceCount=1] >>> >>> TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG >>> {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group >>> monitor is running: [group] g-sc-G3-1-1x0 >>> >>> TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG >>> {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Handling >>> group scaling for the [group] g-sc-G3-1-1x0upon a max out event from the >>> children >>> >>> TID: [0] [STRATOS] [2015-05-26 02:07:51,629] WARN >>> {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - [Group] >>> g-sc-G3-1-1x0 has reached the maximum limit as [max] 4. Hence trying to >>> notify the parent. >>> >>> TID: [0] [STRATOS] [2015-05-26 02:07:51,630] DEBUG >>> {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} - >>> Child Scaling max out event received to [group]: g-sc-G2-1-0x0, [network >>> partition]: RegionOne, [event] g-sc-G12-1, [group instance] g-sc-G2-1-0x0-1 >>> >>> “ >>> >>> >>> >>> I checked the code and added some additional traces and it turns out >>> that the following code is executed: >>> >>> >>> >>> In *GroupMonitor.java-> createInstanceOnDemand()*, with *partitionContext >>> as being null* >>> >>> …. >>> >>> *if (partitionContext != null) {* >>> >>> * groupInstanceId = >>> createGroupInstanceAndAddToMonitor(group, parentInstanceContext,* >>> >>> * partitionContext,* >>> >>> * parentLevelNetworkPartitionContext,* >>> >>> * null);* >>> >>> * instanceIdsToStart.add(groupInstanceId);* >>> >>> * } else {* >>> >>> * log.warn("[Group] " + group.getUniqueIdentifier() + >>> " has reached " +* >>> >>> * "the maximum limit as [max] " + groupMax +* >>> >>> * ". Hence trying to notify the parent.");* >>> >>> * }* >>> >>> >>> >>> >>> >>> Also, no new cartridge instances were created beyond the initial ones. >>> >>> >>> >>> The same “patterns” seems to hold for all other groups within the >>> application, hence no new group instance seems to be created (beyond the >>> initial ones). >>> >>> >>> >>> I am not sure if I am hitting a configuration issue or an issue with >>> group scaling. >>> >>> Please see logs (wso2carbon-8.log debug enabled and artifacts (polices, >>> application.json, cartridge-group.json)) in the attached zip file. >>> >>> >>> >>> At this point I don’t think this is a configuration issue but rather a >>> system issue (please advise if otherwise) , and I will open a JIRA >>> >>> >>> >>> Thanks >>> >>> >>> >>> Martin >>> >>> >>> >>> >>> >>> [1.] application >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> *From:* Martin Eppel (meppel) >>> *Sent:* Friday, May 22, 2015 4:42 PM >>> *To:* dev@stratos.apache.org >>> *Subject:* RE: Testing stratos 4.1: group scaling >>> >>> >>> >>> Hi Reka, >>> >>> >>> >>> I created the following application – see [1.]. >>> >>> >>> >>> I am able to force the autoscaler to decide to scale up but something in >>> my configuration doesn’t seem to be right since it seems to have reached >>> the maximum of instances. I attached the autoscaling policies, deployment >>> policies, as well as cartridge-group.json, application.json and >>> wso2carbon.log to the email. Any pointers would be highly appreciated to >>> what’s missing or incorrectly configured >>> >>> >>> >>> Thanks >>> >>> >>> >>> Martin >>> >>> >>> >>> >>> >>> TID: [0] [STRATOS] [2015-05-22 23:03:14,865] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scale-up] Trying to scale >>> up over max, hence not scaling up cluster itself and >>> >>> notifying to parent for possible group scaling >>> or app bursting. >>> >>> [cluster] g-sc-G12-1.c3-1x0.c3.domain [instance >>> id]g-sc-G2-1-0x0-1 [max] 1 >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> [1.] application >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> *From:* Reka Thirunavukkarasu [mailto:r...@wso2.com <r...@wso2.com>] >>> *Sent:* Friday, May 22, 2015 10:41 AM >>> *To:* dev >>> *Subject:* Re: Testing stratos 4.1: group scaling >>> >>> >>> >>> Hi Martin, >>> >>> If you set the threshold of load average in the autoscaling policy to >>> like 20-40 range and use the below script[1] to stress a particular VM/Vms, >>> then the CPU usage will get higher. In that case, it will eventually exceed >>> the threshold and then stratos will try to scaleup. >>> >>> If you are using memory consumption, then you can try to increase the >>> memory usage using any of your scenarios and make the stratos to scaleup. >>> >>> However, if you set the threshold as very lower values, then even at the >>> beginning stratos will scaleup as the stats will hit the threshold. >>> >>> [1] >>> https://github.com/lahirus/SetupStratos/blob/master/stress/stress_yes.sh >>> >>> Thanks, >>> >>> Reka >>> >>> >>> >>> On Fri, May 22, 2015 at 10:01 PM, Martin Eppel (meppel) < >>> mep...@cisco.com> wrote: >>> >>> Hi Reka, >>> >>> >>> >>> Thanks for the explanation. I think my question was more directed >>> towards how can I simulate the statistics to exceed the thresholds to >>> trigger the scale up, especially in the context of test automation. >>> >>> >>> >>> So to rephrase it, how can I simulate cartridge statistics events >>> without having to modify the cartridge code ? >>> >>> >>> >>> Thanks >>> >>> >>> >>> Martin >>> >>> >>> >>> *From:* Reka Thirunavukkarasu [mailto:r...@wso2.com] >>> *Sent:* Thursday, May 21, 2015 9:59 PM >>> *To:* dev >>> *Subject:* Re: Testing stratos 4.1: group scaling >>> >>> >>> >>> Hi Martin, >>> >>> Let me explain how group scaling works in a high level manner. Say, it a >>> group(G1) has two independent clusters(no dependent scaling) C1 and C2. >>> >>> G1 - min 1 max5 >>> >>> C1 - min 1 max2 (applies per cluster instance) >>> >>> C2 - min 2 max 3 (applies per cluster instance) >>> >>> Where C1 and C2 are children of G1. At the beginning, we will create one >>> group instance say G11 and corresponding cluster instance for C1 and C2 of >>> G11 will get created. Then each C1 and C2's cluster instance will receive >>> stats per cluster instance wise. If the stats is greater than the >>> threshold, then as C1 and C2's cluster instance have room to scaleup by one >>> more instance according min and max configuration, each clusters will get >>> scaleup until it hits the max. Once max is reached, if the stats is still >>> high, then when C1 or C2 doesn't have space to scaleup, it will notify the >>> parent (G1). So, G1 will check whether it has group scaling capability. If >>> so, it will try to create new group instance. If no group scaling enable, >>> it will notify its parent. If the eventual parent application, then >>> application will try to get busted if there is space. >>> >>> If you mention your configuration as below, then when the stats is >>> higher, group scaling will immediately happen as C1 and C2 don't have space >>> to scaleup: >>> >>> G1 - min 1 max5 >>> C1 - min 1 max1 >>> C2 - min 2 max 2 >>> >>> Hope above points will help you to test the group scaling. >>> >>> >>> >>> Thanks, >>> >>> Reka >>> >>> >>> >>> >>> >>> On Fri, May 22, 2015 at 5:20 AM, Martin Eppel (meppel) <mep...@cisco.com> >>> wrote: >>> >>> Hi, >>> >>> >>> >>> I am setting up a test application to test group scaling (following the >>> example from the 4.1 Wiki - >>> https://cwiki.apache.org/confluence/display/STRATOS/4.1.0+Group+Scaling+Application+on+OpenStack). >>> >>> >>> >>> >>> I was wondering if you have any suggestion how to actually force / >>> simulate the conditions to force a group to scale up ? How does it work in >>> the sample application (to get the group to scale up) ? >>> >>> >>> >>> Btw I am using open stack as IaaS, not the Mock IaaS. >>> >>> >>> >>> Thanks >>> >>> >>> >>> Martin >>> >>> >>> >>> >>> >>> >>> -- >>> >>> Reka Thirunavukkarasu >>> Senior Software Engineer, >>> WSO2, Inc.:http://wso2.com, >>> >>> Mobile: +94776442007 >>> >>> >>> >>> >>> >>> >>> -- >>> >>> Reka Thirunavukkarasu >>> Senior Software Engineer, >>> WSO2, Inc.:http://wso2.com, >>> >>> Mobile: +94776442007 >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> Lakmal Warusawithana >>> >>> Vice President, Apache Stratos >>> >>> Director - Cloud Architecture; WSO2 Inc. >>> >>> Mobile : +94714289692 >>> >>> Blog : http://lakmalsview.blogspot.com/ >>> >> >> >> >> -- >> Lakmal Warusawithana >> Vice President, Apache Stratos >> Director - Cloud Architecture; WSO2 Inc. >> Mobile : +94714289692 >> Blog : http://lakmalsview.blogspot.com/ >> >> > > > -- > > Udara Liyanage > Software Engineer > WSO2, Inc.: http://wso2.com > lean. enterprise. middleware > > web: http://udaraliyanage.wordpress.com > phone: +94 71 443 6897 > -- Udara Liyanage Software Engineer WSO2, Inc.: http://wso2.com lean. enterprise. middleware web: http://udaraliyanage.wordpress.com phone: +94 71 443 6897
