Hi Martin, I will try your scenario in Mock IaaS and let you know the results
On Thu, May 28, 2015 at 7:49 AM, Lakmal Warusawithana <lak...@wso2.com> wrote: > Hi Martin > > On Thu, May 28, 2015 at 7:07 AM, Martin Eppel (meppel) <mep...@cisco.com> > wrote: > >> Hi Lakmal, Udara, >> >> >> >> I think I found the issue – apparently for group scaling to work a >> deployment-policy has to be defined at group level (can you confirm this, >> are there special requirements for the group level deployment policies ?). >> >> >> > > Yes, same as individual cartridge scaling, we have to define where to > scale up etc. Since we need group scaling, we need to define deployment > policy in group level to say where to scale up the group. > > >> Once I added it, groups started to scale and create respective VM >> instances. I am still analyzing the logs but I think I am good for the >> moment, >> >> >> >> Thanks >> >> >> >> Martin >> >> >> >> *From:* Martin Eppel (meppel) >> *Sent:* Wednesday, May 27, 2015 9:06 AM >> *To:* dev@stratos.apache.org; Udara Liyanage >> >> *Subject:* RE: Testing stratos 4.1: group scaling issue >> >> >> >> Thanks Lakmal, Udara >> >> >> >> *From:* Lakmal Warusawithana [mailto:lak...@wso2.com <lak...@wso2.com>] >> *Sent:* Tuesday, May 26, 2015 11:00 PM >> *To:* dev@stratos.apache.org; Udara Liyanage >> *Subject:* Re: Testing stratos 4.1: group scaling issue >> >> >> >> Hi Udara, >> >> >> >> if you have some cycles, can you work with Martin on this issue. >> >> >> >> On Wed, May 27, 2015 at 6:02 AM, Martin Eppel (meppel) <mep...@cisco.com> >> wrote: >> >> Attaching the proper deployment-policy-1.json, please disregard the one >> in the zip file >> >> >> >> Thanks >> >> >> >> Martin >> >> >> >> *From:* Martin Eppel (meppel) >> *Sent:* Tuesday, May 26, 2015 5:04 PM >> *To:* dev@stratos.apache.org >> *Subject:* RE: Testing stratos 4.1: group scaling issue >> >> >> >> >> >> Hi Reka, >> >> >> >> I tried various way to get the group scaling to work, so far without >> success. >> >> The latest application I created is as shown below [1.] >> >> >> >> This is what I observe: >> >> >> >> The application starts up normally, with all (initial) cartridges going >> active in the proper sequence >> >> >> >> From the logs: >> >> The very first time a scaling decision is made to scale up a group it >> seems form the log that the max number is already reached: >> >> “ >> >> TID: [0] [STRATOS] [2015-05-26 02:07:51,628] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scale-up] Trying to scale >> up over max, hence not scaling up cluster itself and >> >> notifying to parent for possible group scaling or >> app bursting. >> >> [cluster] g-sc-G12-1.c5-2x0.c5.domain [instance >> id]g-sc-G3-1-1x0-1 [max] 1 >> >> ” >> >> >> >> It is followed by a subsequent log entry complaining that the max limit >> of 4 has been reached. I am assuming this refers back to the >> “groupMaxInstances” in the application.json of group with alias "g-sc-G3-1". >> >> >> >> However, I don’t see any entries which indicate that group instances were >> successfully created before hitting the max number. >> >> >> >> “ >> >> TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling max out >> notification is going to the [parentInstance] g-sc-G3-1-1x0-1 >> >> TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG >> {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} - >> Child Scaling max out event received to [group]: g-sc-G3-1-1x0, [network >> partition]: RegionOne, [event] g-sc-G12-1.c5-2x0.c5.domain, [group >> instance] g-sc-G3-1-1x0-1 >> >> TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG >> {org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor} - Rule >> executed for: NetworkPartitionContext >> [id=g-sc-G3-1-1x0-1partitionAlgorithm=one-after-another, >> minInstanceCount=1, maxInstanceCount=1] >> >> TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG >> {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group >> monitor is running: [group] g-sc-G3-1-1x0 >> >> TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG >> {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Handling >> group scaling for the [group] g-sc-G3-1-1x0upon a max out event from the >> children >> >> TID: [0] [STRATOS] [2015-05-26 02:07:51,629] WARN >> {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - [Group] >> g-sc-G3-1-1x0 has reached the maximum limit as [max] 4. Hence trying to >> notify the parent. >> >> TID: [0] [STRATOS] [2015-05-26 02:07:51,630] DEBUG >> {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} - >> Child Scaling max out event received to [group]: g-sc-G2-1-0x0, [network >> partition]: RegionOne, [event] g-sc-G12-1, [group instance] g-sc-G2-1-0x0-1 >> >> “ >> >> >> >> I checked the code and added some additional traces and it turns out that >> the following code is executed: >> >> >> >> In *GroupMonitor.java-> createInstanceOnDemand()*, with *partitionContext >> as being null* >> >> …. >> >> *if (partitionContext != null) {* >> >> * groupInstanceId = >> createGroupInstanceAndAddToMonitor(group, parentInstanceContext,* >> >> * partitionContext,* >> >> * parentLevelNetworkPartitionContext,* >> >> * null);* >> >> * instanceIdsToStart.add(groupInstanceId);* >> >> * } else {* >> >> * log.warn("[Group] " + group.getUniqueIdentifier() + >> " has reached " +* >> >> * "the maximum limit as [max] " + groupMax +* >> >> * ". Hence trying to notify the parent.");* >> >> * }* >> >> >> >> >> >> Also, no new cartridge instances were created beyond the initial ones. >> >> >> >> The same “patterns” seems to hold for all other groups within the >> application, hence no new group instance seems to be created (beyond the >> initial ones). >> >> >> >> I am not sure if I am hitting a configuration issue or an issue with >> group scaling. >> >> Please see logs (wso2carbon-8.log debug enabled and artifacts (polices, >> application.json, cartridge-group.json)) in the attached zip file. >> >> >> >> At this point I don’t think this is a configuration issue but rather a >> system issue (please advise if otherwise) , and I will open a JIRA >> >> >> >> Thanks >> >> >> >> Martin >> >> >> >> >> >> [1.] application >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> *From:* Martin Eppel (meppel) >> *Sent:* Friday, May 22, 2015 4:42 PM >> *To:* dev@stratos.apache.org >> *Subject:* RE: Testing stratos 4.1: group scaling >> >> >> >> Hi Reka, >> >> >> >> I created the following application – see [1.]. >> >> >> >> I am able to force the autoscaler to decide to scale up but something in >> my configuration doesn’t seem to be right since it seems to have reached >> the maximum of instances. I attached the autoscaling policies, deployment >> policies, as well as cartridge-group.json, application.json and >> wso2carbon.log to the email. Any pointers would be highly appreciated to >> what’s missing or incorrectly configured >> >> >> >> Thanks >> >> >> >> Martin >> >> >> >> >> >> TID: [0] [STRATOS] [2015-05-22 23:03:14,865] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scale-up] Trying to scale >> up over max, hence not scaling up cluster itself and >> >> notifying to parent for possible group scaling or >> app bursting. >> >> [cluster] g-sc-G12-1.c3-1x0.c3.domain [instance >> id]g-sc-G2-1-0x0-1 [max] 1 >> >> >> >> >> >> >> >> >> >> [1.] application >> >> >> >> >> >> >> >> >> >> *From:* Reka Thirunavukkarasu [mailto:r...@wso2.com <r...@wso2.com>] >> *Sent:* Friday, May 22, 2015 10:41 AM >> *To:* dev >> *Subject:* Re: Testing stratos 4.1: group scaling >> >> >> >> Hi Martin, >> >> If you set the threshold of load average in the autoscaling policy to >> like 20-40 range and use the below script[1] to stress a particular VM/Vms, >> then the CPU usage will get higher. In that case, it will eventually exceed >> the threshold and then stratos will try to scaleup. >> >> If you are using memory consumption, then you can try to increase the >> memory usage using any of your scenarios and make the stratos to scaleup. >> >> However, if you set the threshold as very lower values, then even at the >> beginning stratos will scaleup as the stats will hit the threshold. >> >> [1] >> https://github.com/lahirus/SetupStratos/blob/master/stress/stress_yes.sh >> >> Thanks, >> >> Reka >> >> >> >> On Fri, May 22, 2015 at 10:01 PM, Martin Eppel (meppel) <mep...@cisco.com> >> wrote: >> >> Hi Reka, >> >> >> >> Thanks for the explanation. I think my question was more directed towards >> how can I simulate the statistics to exceed the thresholds to trigger the >> scale up, especially in the context of test automation. >> >> >> >> So to rephrase it, how can I simulate cartridge statistics events without >> having to modify the cartridge code ? >> >> >> >> Thanks >> >> >> >> Martin >> >> >> >> *From:* Reka Thirunavukkarasu [mailto:r...@wso2.com] >> *Sent:* Thursday, May 21, 2015 9:59 PM >> *To:* dev >> *Subject:* Re: Testing stratos 4.1: group scaling >> >> >> >> Hi Martin, >> >> Let me explain how group scaling works in a high level manner. Say, it a >> group(G1) has two independent clusters(no dependent scaling) C1 and C2. >> >> G1 - min 1 max5 >> >> C1 - min 1 max2 (applies per cluster instance) >> >> C2 - min 2 max 3 (applies per cluster instance) >> >> Where C1 and C2 are children of G1. At the beginning, we will create one >> group instance say G11 and corresponding cluster instance for C1 and C2 of >> G11 will get created. Then each C1 and C2's cluster instance will receive >> stats per cluster instance wise. If the stats is greater than the >> threshold, then as C1 and C2's cluster instance have room to scaleup by one >> more instance according min and max configuration, each clusters will get >> scaleup until it hits the max. Once max is reached, if the stats is still >> high, then when C1 or C2 doesn't have space to scaleup, it will notify the >> parent (G1). So, G1 will check whether it has group scaling capability. If >> so, it will try to create new group instance. If no group scaling enable, >> it will notify its parent. If the eventual parent application, then >> application will try to get busted if there is space. >> >> If you mention your configuration as below, then when the stats is >> higher, group scaling will immediately happen as C1 and C2 don't have space >> to scaleup: >> >> G1 - min 1 max5 >> C1 - min 1 max1 >> C2 - min 2 max 2 >> >> Hope above points will help you to test the group scaling. >> >> >> >> Thanks, >> >> Reka >> >> >> >> >> >> On Fri, May 22, 2015 at 5:20 AM, Martin Eppel (meppel) <mep...@cisco.com> >> wrote: >> >> Hi, >> >> >> >> I am setting up a test application to test group scaling (following the >> example from the 4.1 Wiki - >> https://cwiki.apache.org/confluence/display/STRATOS/4.1.0+Group+Scaling+Application+on+OpenStack). >> >> >> >> >> I was wondering if you have any suggestion how to actually force / >> simulate the conditions to force a group to scale up ? How does it work in >> the sample application (to get the group to scale up) ? >> >> >> >> Btw I am using open stack as IaaS, not the Mock IaaS. >> >> >> >> Thanks >> >> >> >> Martin >> >> >> >> >> >> >> -- >> >> Reka Thirunavukkarasu >> Senior Software Engineer, >> WSO2, Inc.:http://wso2.com, >> >> Mobile: +94776442007 >> >> >> >> >> >> >> -- >> >> Reka Thirunavukkarasu >> Senior Software Engineer, >> WSO2, Inc.:http://wso2.com, >> >> Mobile: +94776442007 >> >> >> >> >> >> >> >> -- >> >> Lakmal Warusawithana >> >> Vice President, Apache Stratos >> >> Director - Cloud Architecture; WSO2 Inc. >> >> Mobile : +94714289692 >> >> Blog : http://lakmalsview.blogspot.com/ >> > > > > -- > Lakmal Warusawithana > Vice President, Apache Stratos > Director - Cloud Architecture; WSO2 Inc. > Mobile : +94714289692 > Blog : http://lakmalsview.blogspot.com/ > > -- Udara Liyanage Software Engineer WSO2, Inc.: http://wso2.com lean. enterprise. middleware web: http://udaraliyanage.wordpress.com phone: +94 71 443 6897
