Hi Martin On Thu, May 28, 2015 at 7:07 AM, Martin Eppel (meppel) <[email protected]> wrote:
> Hi Lakmal, Udara, > > > > I think I found the issue – apparently for group scaling to work a > deployment-policy has to be defined at group level (can you confirm this, > are there special requirements for the group level deployment policies ?). > > > Yes, same as individual cartridge scaling, we have to define where to scale up etc. Since we need group scaling, we need to define deployment policy in group level to say where to scale up the group. > Once I added it, groups started to scale and create respective VM > instances. I am still analyzing the logs but I think I am good for the > moment, > > > > Thanks > > > > Martin > > > > *From:* Martin Eppel (meppel) > *Sent:* Wednesday, May 27, 2015 9:06 AM > *To:* [email protected]; Udara Liyanage > > *Subject:* RE: Testing stratos 4.1: group scaling issue > > > > Thanks Lakmal, Udara > > > > *From:* Lakmal Warusawithana [mailto:[email protected] <[email protected]>] > *Sent:* Tuesday, May 26, 2015 11:00 PM > *To:* [email protected]; Udara Liyanage > *Subject:* Re: Testing stratos 4.1: group scaling issue > > > > Hi Udara, > > > > if you have some cycles, can you work with Martin on this issue. > > > > On Wed, May 27, 2015 at 6:02 AM, Martin Eppel (meppel) <[email protected]> > wrote: > > Attaching the proper deployment-policy-1.json, please disregard the one in > the zip file > > > > Thanks > > > > Martin > > > > *From:* Martin Eppel (meppel) > *Sent:* Tuesday, May 26, 2015 5:04 PM > *To:* [email protected] > *Subject:* RE: Testing stratos 4.1: group scaling issue > > > > > > Hi Reka, > > > > I tried various way to get the group scaling to work, so far without > success. > > The latest application I created is as shown below [1.] > > > > This is what I observe: > > > > The application starts up normally, with all (initial) cartridges going > active in the proper sequence > > > > From the logs: > > The very first time a scaling decision is made to scale up a group it > seems form the log that the max number is already reached: > > “ > > TID: [0] [STRATOS] [2015-05-26 02:07:51,628] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scale-up] Trying to scale > up over max, hence not scaling up cluster itself and > > notifying to parent for possible group scaling or > app bursting. > > [cluster] g-sc-G12-1.c5-2x0.c5.domain [instance > id]g-sc-G3-1-1x0-1 [max] 1 > > ” > > > > It is followed by a subsequent log entry complaining that the max limit > of 4 has been reached. I am assuming this refers back to the > “groupMaxInstances” in the application.json of group with alias "g-sc-G3-1". > > > > However, I don’t see any entries which indicate that group instances were > successfully created before hitting the max number. > > > > “ > > TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling max out > notification is going to the [parentInstance] g-sc-G3-1-1x0-1 > > TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG > {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} - > Child Scaling max out event received to [group]: g-sc-G3-1-1x0, [network > partition]: RegionOne, [event] g-sc-G12-1.c5-2x0.c5.domain, [group > instance] g-sc-G3-1-1x0-1 > > TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG > {org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor} - Rule > executed for: NetworkPartitionContext > [id=g-sc-G3-1-1x0-1partitionAlgorithm=one-after-another, > minInstanceCount=1, maxInstanceCount=1] > > TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG > {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group > monitor is running: [group] g-sc-G3-1-1x0 > > TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG > {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Handling > group scaling for the [group] g-sc-G3-1-1x0upon a max out event from the > children > > TID: [0] [STRATOS] [2015-05-26 02:07:51,629] WARN > {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - [Group] > g-sc-G3-1-1x0 has reached the maximum limit as [max] 4. Hence trying to > notify the parent. > > TID: [0] [STRATOS] [2015-05-26 02:07:51,630] DEBUG > {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} - > Child Scaling max out event received to [group]: g-sc-G2-1-0x0, [network > partition]: RegionOne, [event] g-sc-G12-1, [group instance] g-sc-G2-1-0x0-1 > > “ > > > > I checked the code and added some additional traces and it turns out that > the following code is executed: > > > > In *GroupMonitor.java-> createInstanceOnDemand()*, with *partitionContext > as being null* > > …. > > *if (partitionContext != null) {* > > * groupInstanceId = > createGroupInstanceAndAddToMonitor(group, parentInstanceContext,* > > * partitionContext,* > > * parentLevelNetworkPartitionContext,* > > * null);* > > * instanceIdsToStart.add(groupInstanceId);* > > * } else {* > > * log.warn("[Group] " + group.getUniqueIdentifier() + " > has reached " +* > > * "the maximum limit as [max] " + groupMax +* > > * ". Hence trying to notify the parent.");* > > * }* > > > > > > Also, no new cartridge instances were created beyond the initial ones. > > > > The same “patterns” seems to hold for all other groups within the > application, hence no new group instance seems to be created (beyond the > initial ones). > > > > I am not sure if I am hitting a configuration issue or an issue with group > scaling. > > Please see logs (wso2carbon-8.log debug enabled and artifacts (polices, > application.json, cartridge-group.json)) in the attached zip file. > > > > At this point I don’t think this is a configuration issue but rather a > system issue (please advise if otherwise) , and I will open a JIRA > > > > Thanks > > > > Martin > > > > > > [1.] application > > > > > > > > > > > > > > > > *From:* Martin Eppel (meppel) > *Sent:* Friday, May 22, 2015 4:42 PM > *To:* [email protected] > *Subject:* RE: Testing stratos 4.1: group scaling > > > > Hi Reka, > > > > I created the following application – see [1.]. > > > > I am able to force the autoscaler to decide to scale up but something in > my configuration doesn’t seem to be right since it seems to have reached > the maximum of instances. I attached the autoscaling policies, deployment > policies, as well as cartridge-group.json, application.json and > wso2carbon.log to the email. Any pointers would be highly appreciated to > what’s missing or incorrectly configured > > > > Thanks > > > > Martin > > > > > > TID: [0] [STRATOS] [2015-05-22 23:03:14,865] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scale-up] Trying to scale > up over max, hence not scaling up cluster itself and > > notifying to parent for possible group scaling or > app bursting. > > [cluster] g-sc-G12-1.c3-1x0.c3.domain [instance > id]g-sc-G2-1-0x0-1 [max] 1 > > > > > > > > > > [1.] application > > > > > > > > > > *From:* Reka Thirunavukkarasu [mailto:[email protected] <[email protected]>] > *Sent:* Friday, May 22, 2015 10:41 AM > *To:* dev > *Subject:* Re: Testing stratos 4.1: group scaling > > > > Hi Martin, > > If you set the threshold of load average in the autoscaling policy to like > 20-40 range and use the below script[1] to stress a particular VM/Vms, then > the CPU usage will get higher. In that case, it will eventually exceed the > threshold and then stratos will try to scaleup. > > If you are using memory consumption, then you can try to increase the > memory usage using any of your scenarios and make the stratos to scaleup. > > However, if you set the threshold as very lower values, then even at the > beginning stratos will scaleup as the stats will hit the threshold. > > [1] > https://github.com/lahirus/SetupStratos/blob/master/stress/stress_yes.sh > > Thanks, > > Reka > > > > On Fri, May 22, 2015 at 10:01 PM, Martin Eppel (meppel) <[email protected]> > wrote: > > Hi Reka, > > > > Thanks for the explanation. I think my question was more directed towards > how can I simulate the statistics to exceed the thresholds to trigger the > scale up, especially in the context of test automation. > > > > So to rephrase it, how can I simulate cartridge statistics events without > having to modify the cartridge code ? > > > > Thanks > > > > Martin > > > > *From:* Reka Thirunavukkarasu [mailto:[email protected]] > *Sent:* Thursday, May 21, 2015 9:59 PM > *To:* dev > *Subject:* Re: Testing stratos 4.1: group scaling > > > > Hi Martin, > > Let me explain how group scaling works in a high level manner. Say, it a > group(G1) has two independent clusters(no dependent scaling) C1 and C2. > > G1 - min 1 max5 > > C1 - min 1 max2 (applies per cluster instance) > > C2 - min 2 max 3 (applies per cluster instance) > > Where C1 and C2 are children of G1. At the beginning, we will create one > group instance say G11 and corresponding cluster instance for C1 and C2 of > G11 will get created. Then each C1 and C2's cluster instance will receive > stats per cluster instance wise. If the stats is greater than the > threshold, then as C1 and C2's cluster instance have room to scaleup by one > more instance according min and max configuration, each clusters will get > scaleup until it hits the max. Once max is reached, if the stats is still > high, then when C1 or C2 doesn't have space to scaleup, it will notify the > parent (G1). So, G1 will check whether it has group scaling capability. If > so, it will try to create new group instance. If no group scaling enable, > it will notify its parent. If the eventual parent application, then > application will try to get busted if there is space. > > If you mention your configuration as below, then when the stats is higher, > group scaling will immediately happen as C1 and C2 don't have space to > scaleup: > > G1 - min 1 max5 > C1 - min 1 max1 > C2 - min 2 max 2 > > Hope above points will help you to test the group scaling. > > > > Thanks, > > Reka > > > > > > On Fri, May 22, 2015 at 5:20 AM, Martin Eppel (meppel) <[email protected]> > wrote: > > Hi, > > > > I am setting up a test application to test group scaling (following the > example from the 4.1 Wiki - > https://cwiki.apache.org/confluence/display/STRATOS/4.1.0+Group+Scaling+Application+on+OpenStack). > > > > > I was wondering if you have any suggestion how to actually force / > simulate the conditions to force a group to scale up ? How does it work in > the sample application (to get the group to scale up) ? > > > > Btw I am using open stack as IaaS, not the Mock IaaS. > > > > Thanks > > > > Martin > > > > > > > -- > > Reka Thirunavukkarasu > Senior Software Engineer, > WSO2, Inc.:http://wso2.com, > > Mobile: +94776442007 > > > > > > > -- > > Reka Thirunavukkarasu > Senior Software Engineer, > WSO2, Inc.:http://wso2.com, > > Mobile: +94776442007 > > > > > > > > -- > > Lakmal Warusawithana > > Vice President, Apache Stratos > > Director - Cloud Architecture; WSO2 Inc. > > Mobile : +94714289692 > > Blog : http://lakmalsview.blogspot.com/ > -- Lakmal Warusawithana Vice President, Apache Stratos Director - Cloud Architecture; WSO2 Inc. Mobile : +94714289692 Blog : http://lakmalsview.blogspot.com/
