Attaching the proper deployment-policy-1.json, please disregard the one in the zip file
Thanks Martin From: Martin Eppel (meppel) Sent: Tuesday, May 26, 2015 5:04 PM To: dev@stratos.apache.org Subject: RE: Testing stratos 4.1: group scaling issue Hi Reka, I tried various way to get the group scaling to work, so far without success. The latest application I created is as shown below [1.] This is what I observe: The application starts up normally, with all (initial) cartridges going active in the proper sequence From the logs: The very first time a scaling decision is made to scale up a group it seems form the log that the max number is already reached: “ TID: [0] [STRATOS] [2015-05-26 02:07:51,628] INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scale-up] Trying to scale up over max, hence not scaling up cluster itself and notifying to parent for possible group scaling or app bursting. [cluster] g-sc-G12-1.c5-2x0.c5.domain [instance id]g-sc-G3-1-1x0-1 [max] 1 ” It is followed by a subsequent log entry complaining that the max limit of 4 has been reached. I am assuming this refers back to the “groupMaxInstances” in the application.json of group with alias "g-sc-G3-1". However, I don’t see any entries which indicate that group instances were successfully created before hitting the max number. “ TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling max out notification is going to the [parentInstance] g-sc-G3-1-1x0-1 TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} - Child Scaling max out event received to [group]: g-sc-G3-1-1x0, [network partition]: RegionOne, [event] g-sc-G12-1.c5-2x0.c5.domain, [group instance] g-sc-G3-1-1x0-1 TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG {org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor} - Rule executed for: NetworkPartitionContext [id=g-sc-G3-1-1x0-1partitionAlgorithm=one-after-another, minInstanceCount=1, maxInstanceCount=1] TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group monitor is running: [group] g-sc-G3-1-1x0 TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Handling group scaling for the [group] g-sc-G3-1-1x0upon a max out event from the children TID: [0] [STRATOS] [2015-05-26 02:07:51,629] WARN {org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - [Group] g-sc-G3-1-1x0 has reached the maximum limit as [max] 4. Hence trying to notify the parent. TID: [0] [STRATOS] [2015-05-26 02:07:51,630] DEBUG {org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} - Child Scaling max out event received to [group]: g-sc-G2-1-0x0, [network partition]: RegionOne, [event] g-sc-G12-1, [group instance] g-sc-G2-1-0x0-1 “ I checked the code and added some additional traces and it turns out that the following code is executed: In GroupMonitor.java-> createInstanceOnDemand(), with partitionContext as being null …. if (partitionContext != null) { groupInstanceId = createGroupInstanceAndAddToMonitor(group, parentInstanceContext, partitionContext, parentLevelNetworkPartitionContext, null); instanceIdsToStart.add(groupInstanceId); } else { log.warn("[Group] " + group.getUniqueIdentifier() + " has reached " + "the maximum limit as [max] " + groupMax + ". Hence trying to notify the parent."); } Also, no new cartridge instances were created beyond the initial ones. The same “patterns” seems to hold for all other groups within the application, hence no new group instance seems to be created (beyond the initial ones). I am not sure if I am hitting a configuration issue or an issue with group scaling. Please see logs (wso2carbon-8.log debug enabled and artifacts (polices, application.json, cartridge-group.json)) in the attached zip file. At this point I don’t think this is a configuration issue but rather a system issue (please advise if otherwise) , and I will open a JIRA Thanks Martin [1.] application [cid:image002.png@01D097D9.FA9FD2E0] From: Martin Eppel (meppel) Sent: Friday, May 22, 2015 4:42 PM To: dev@stratos.apache.org<mailto:dev@stratos.apache.org> Subject: RE: Testing stratos 4.1: group scaling Hi Reka, I created the following application – see [1.]. I am able to force the autoscaler to decide to scale up but something in my configuration doesn’t seem to be right since it seems to have reached the maximum of instances. I attached the autoscaling policies, deployment policies, as well as cartridge-group.json, application.json and wso2carbon.log to the email. Any pointers would be highly appreciated to what’s missing or incorrectly configured Thanks Martin TID: [0] [STRATOS] [2015-05-22 23:03:14,865] INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scale-up] Trying to scale up over max, hence not scaling up cluster itself and notifying to parent for possible group scaling or app bursting. [cluster] g-sc-G12-1.c3-1x0.c3.domain [instance id]g-sc-G2-1-0x0-1 [max] 1 [1.] application [cid:image003.png@01D097D9.FA9FD2E0] From: Reka Thirunavukkarasu [mailto:r...@wso2.com] Sent: Friday, May 22, 2015 10:41 AM To: dev Subject: Re: Testing stratos 4.1: group scaling Hi Martin, If you set the threshold of load average in the autoscaling policy to like 20-40 range and use the below script[1] to stress a particular VM/Vms, then the CPU usage will get higher. In that case, it will eventually exceed the threshold and then stratos will try to scaleup. If you are using memory consumption, then you can try to increase the memory usage using any of your scenarios and make the stratos to scaleup. However, if you set the threshold as very lower values, then even at the beginning stratos will scaleup as the stats will hit the threshold. [1] https://github.com/lahirus/SetupStratos/blob/master/stress/stress_yes.sh Thanks, Reka On Fri, May 22, 2015 at 10:01 PM, Martin Eppel (meppel) <mep...@cisco.com<mailto:mep...@cisco.com>> wrote: Hi Reka, Thanks for the explanation. I think my question was more directed towards how can I simulate the statistics to exceed the thresholds to trigger the scale up, especially in the context of test automation. So to rephrase it, how can I simulate cartridge statistics events without having to modify the cartridge code ? Thanks Martin From: Reka Thirunavukkarasu [mailto:r...@wso2.com<mailto:r...@wso2.com>] Sent: Thursday, May 21, 2015 9:59 PM To: dev Subject: Re: Testing stratos 4.1: group scaling Hi Martin, Let me explain how group scaling works in a high level manner. Say, it a group(G1) has two independent clusters(no dependent scaling) C1 and C2. G1 - min 1 max5 C1 - min 1 max2 (applies per cluster instance) C2 - min 2 max 3 (applies per cluster instance) Where C1 and C2 are children of G1. At the beginning, we will create one group instance say G11 and corresponding cluster instance for C1 and C2 of G11 will get created. Then each C1 and C2's cluster instance will receive stats per cluster instance wise. If the stats is greater than the threshold, then as C1 and C2's cluster instance have room to scaleup by one more instance according min and max configuration, each clusters will get scaleup until it hits the max. Once max is reached, if the stats is still high, then when C1 or C2 doesn't have space to scaleup, it will notify the parent (G1). So, G1 will check whether it has group scaling capability. If so, it will try to create new group instance. If no group scaling enable, it will notify its parent. If the eventual parent application, then application will try to get busted if there is space. If you mention your configuration as below, then when the stats is higher, group scaling will immediately happen as C1 and C2 don't have space to scaleup: G1 - min 1 max5 C1 - min 1 max1 C2 - min 2 max 2 Hope above points will help you to test the group scaling. Thanks, Reka On Fri, May 22, 2015 at 5:20 AM, Martin Eppel (meppel) <mep...@cisco.com<mailto:mep...@cisco.com>> wrote: Hi, I am setting up a test application to test group scaling (following the example from the 4.1 Wiki - https://cwiki.apache.org/confluence/display/STRATOS/4.1.0+Group+Scaling+Application+on+OpenStack). I was wondering if you have any suggestion how to actually force / simulate the conditions to force a group to scale up ? How does it work in the sample application (to get the group to scale up) ? Btw I am using open stack as IaaS, not the Mock IaaS. Thanks Martin -- Reka Thirunavukkarasu Senior Software Engineer, WSO2, Inc.:http://wso2.com, Mobile: +94776442007<tel:%2B94776442007> -- Reka Thirunavukkarasu Senior Software Engineer, WSO2, Inc.:http://wso2.com, Mobile: +94776442007
deployment-policy-1.json
Description: deployment-policy-1.json