Attaching the proper deployment-policy-1.json, please disregard the one in the zip file
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Tuesday, May 26, 2015 5:04 PM
To: dev@stratos.apache.org
Subject: RE: Testing stratos 4.1: group scaling issue
Hi Reka,
I tried various way to get the group scaling to work, so far without success.
The latest application I created is as shown below [1.]
This is what I observe:
The application starts up normally, with all (initial) cartridges going active
in the proper sequence
From the logs:
The very first time a scaling decision is made to scale up a group it seems
form the log that the max number is already reached:
“
TID: [0] [STRATOS] [2015-05-26 02:07:51,628] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scale-up] Trying to scale up
over max, hence not scaling up cluster itself and
notifying to parent for possible group scaling or app
bursting.
[cluster] g-sc-G12-1.c5-2x0.c5.domain [instance
id]g-sc-G3-1-1x0-1 [max] 1
”
It is followed by a subsequent log entry complaining that the max limit of 4
has been reached. I am assuming this refers back to the “groupMaxInstances” in
the application.json of group with alias "g-sc-G3-1".
However, I don’t see any entries which indicate that group instances were
successfully created before hitting the max number.
“
TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG
{org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling max out
notification is going to the [parentInstance] g-sc-G3-1-1x0-1
TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG
{org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} -
Child Scaling max out event received to [group]: g-sc-G3-1-1x0, [network
partition]: RegionOne, [event] g-sc-G12-1.c5-2x0.c5.domain, [group instance]
g-sc-G3-1-1x0-1
TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG
{org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor} - Rule executed
for: NetworkPartitionContext
[id=g-sc-G3-1-1x0-1partitionAlgorithm=one-after-another, minInstanceCount=1,
maxInstanceCount=1]
TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG
{org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Group monitor
is running: [group] g-sc-G3-1-1x0
TID: [0] [STRATOS] [2015-05-26 02:07:51,629] DEBUG
{org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - Handling
group scaling for the [group] g-sc-G3-1-1x0upon a max out event from the
children
TID: [0] [STRATOS] [2015-05-26 02:07:51,629] WARN
{org.apache.stratos.autoscaler.monitor.component.GroupMonitor} - [Group]
g-sc-G3-1-1x0 has reached the maximum limit as [max] 4. Hence trying to notify
the parent.
TID: [0] [STRATOS] [2015-05-26 02:07:51,630] DEBUG
{org.apache.stratos.autoscaler.monitor.component.ParentComponentMonitor} -
Child Scaling max out event received to [group]: g-sc-G2-1-0x0, [network
partition]: RegionOne, [event] g-sc-G12-1, [group instance] g-sc-G2-1-0x0-1
“
I checked the code and added some additional traces and it turns out that the
following code is executed:
In GroupMonitor.java-> createInstanceOnDemand(), with partitionContext as being
null
….
if (partitionContext != null) {
groupInstanceId = createGroupInstanceAndAddToMonitor(group,
parentInstanceContext,
partitionContext,
parentLevelNetworkPartitionContext,
null);
instanceIdsToStart.add(groupInstanceId);
} else {
log.warn("[Group] " + group.getUniqueIdentifier() + " has
reached " +
"the maximum limit as [max] " + groupMax +
". Hence trying to notify the parent.");
}
Also, no new cartridge instances were created beyond the initial ones.
The same “patterns” seems to hold for all other groups within the application,
hence no new group instance seems to be created (beyond the initial ones).
I am not sure if I am hitting a configuration issue or an issue with group
scaling.
Please see logs (wso2carbon-8.log debug enabled and artifacts (polices,
application.json, cartridge-group.json)) in the attached zip file.
At this point I don’t think this is a configuration issue but rather a system
issue (please advise if otherwise) , and I will open a JIRA
Thanks
Martin
[1.] application
[cid:image002.png@01D097D9.FA9FD2E0]
From: Martin Eppel (meppel)
Sent: Friday, May 22, 2015 4:42 PM
To: dev@stratos.apache.org<mailto:dev@stratos.apache.org>
Subject: RE: Testing stratos 4.1: group scaling
Hi Reka,
I created the following application – see [1.].
I am able to force the autoscaler to decide to scale up but something in my
configuration doesn’t seem to be right since it seems to have reached the
maximum of instances. I attached the autoscaling policies, deployment policies,
as well as cartridge-group.json, application.json and wso2carbon.log to the
email. Any pointers would be highly appreciated to what’s missing or
incorrectly configured
Thanks
Martin
TID: [0] [STRATOS] [2015-05-22 23:03:14,865] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scale-up] Trying to scale up
over max, hence not scaling up cluster itself and
notifying to parent for possible group scaling or app
bursting.
[cluster] g-sc-G12-1.c3-1x0.c3.domain [instance
id]g-sc-G2-1-0x0-1 [max] 1
[1.] application
[cid:image003.png@01D097D9.FA9FD2E0]
From: Reka Thirunavukkarasu [mailto:r...@wso2.com]
Sent: Friday, May 22, 2015 10:41 AM
To: dev
Subject: Re: Testing stratos 4.1: group scaling
Hi Martin,
If you set the threshold of load average in the autoscaling policy to like
20-40 range and use the below script[1] to stress a particular VM/Vms, then the
CPU usage will get higher. In that case, it will eventually exceed the
threshold and then stratos will try to scaleup.
If you are using memory consumption, then you can try to increase the memory
usage using any of your scenarios and make the stratos to scaleup.
However, if you set the threshold as very lower values, then even at the
beginning stratos will scaleup as the stats will hit the threshold.
[1] https://github.com/lahirus/SetupStratos/blob/master/stress/stress_yes.sh
Thanks,
Reka
On Fri, May 22, 2015 at 10:01 PM, Martin Eppel (meppel)
<mep...@cisco.com<mailto:mep...@cisco.com>> wrote:
Hi Reka,
Thanks for the explanation. I think my question was more directed towards how
can I simulate the statistics to exceed the thresholds to trigger the scale up,
especially in the context of test automation.
So to rephrase it, how can I simulate cartridge statistics events without
having to modify the cartridge code ?
Thanks
Martin
From: Reka Thirunavukkarasu [mailto:r...@wso2.com<mailto:r...@wso2.com>]
Sent: Thursday, May 21, 2015 9:59 PM
To: dev
Subject: Re: Testing stratos 4.1: group scaling
Hi Martin,
Let me explain how group scaling works in a high level manner. Say, it a
group(G1) has two independent clusters(no dependent scaling) C1 and C2.
G1 - min 1 max5
C1 - min 1 max2 (applies per cluster instance)
C2 - min 2 max 3 (applies per cluster instance)
Where C1 and C2 are children of G1. At the beginning, we will create one group
instance say G11 and corresponding cluster instance for C1 and C2 of G11 will
get created. Then each C1 and C2's cluster instance will receive stats per
cluster instance wise. If the stats is greater than the threshold, then as C1
and C2's cluster instance have room to scaleup by one more instance according
min and max configuration, each clusters will get scaleup until it hits the
max. Once max is reached, if the stats is still high, then when C1 or C2
doesn't have space to scaleup, it will notify the parent (G1). So, G1 will
check whether it has group scaling capability. If so, it will try to create new
group instance. If no group scaling enable, it will notify its parent. If the
eventual parent application, then application will try to get busted if there
is space.
If you mention your configuration as below, then when the stats is higher,
group scaling will immediately happen as C1 and C2 don't have space to scaleup:
G1 - min 1 max5
C1 - min 1 max1
C2 - min 2 max 2
Hope above points will help you to test the group scaling.
Thanks,
Reka
On Fri, May 22, 2015 at 5:20 AM, Martin Eppel (meppel)
<mep...@cisco.com<mailto:mep...@cisco.com>> wrote:
Hi,
I am setting up a test application to test group scaling (following the example
from the 4.1 Wiki -
https://cwiki.apache.org/confluence/display/STRATOS/4.1.0+Group+Scaling+Application+on+OpenStack).
I was wondering if you have any suggestion how to actually force / simulate the
conditions to force a group to scale up ? How does it work in the sample
application (to get the group to scale up) ?
Btw I am using open stack as IaaS, not the Mock IaaS.
Thanks
Martin
--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007<tel:%2B94776442007>
--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007
deployment-policy-1.json
Description: deployment-policy-1.json
