Hi David,
I have few concerns about your deployment policies.
1. I see only 2 deployment policies defined: static-1, autoscale-1-2
But in the cartridge subscription I can see a deployment policy named
autoscale-1-5. Where did that come from? Can you share your complete
deployment policy/policies?
2. In deployment policy autoscale-1-2, you have given the provide as:
"provider":"openstack-Core". Is this correct? Can you share your
cloud-controller.xml?
As Nirmal suggested, enabling DEBUG logs will give more insights into
what is actually causing this issue. It's better if you can enable
DEBUG logs for the autoscaler package to get the complete picture.
log4j.logger.org.apache.stratos.autoscaler=INFO
Thanks.
On Fri, Aug 1, 2014 at 11:30 AM, Nirmal Fernando <nirmal070...@gmail.com> wrote:
>
>
>
> On Fri, Aug 1, 2014 at 11:30 AM, Nirmal Fernando <nirmal070...@gmail.com>
> wrote:
>>
>> Hi David,
>>
>> Is there any possibility of enabling following logger in the
>> log4j.properties file ?
>>
>> log4j.logger.org.apache.stratos.autoscaler.rule.RuleLog=DEBUG
>>
>>
>> For each service cluster, we run a Cluster Monitor and that is responsible
>> for monitoring and scaling the cluster.
>>
>>
>>
>> On Thu, Jul 31, 2014 at 10:56 PM, David Waddell
>> <david.wadd...@owmobility.com> wrote:
>>>
>>> Hi guys
>>>
>>> We’re experiencing an issue on stratos 4.0 - the autoscaler doesn’t
>>> seem to be kicking in when multiple cartridges are subscribed.
>>>
>>> When deploying only one cartridge, the autoscaler works as expected
>>> .
>>>
>>>
>>>
>>> 3 cartridges are defined : opwv-oam-01, opwv-oam-02, opwv-vos.
>>>
>>>
>>>
>>> {"displayName":"opwv-vos","description":"opwv-vos
>>> Cartridge","cartridgeAlias":"-","cartridgeType":"opwv-vos","activeInstances":0,"provider":"cisco","version":"1","multiTenant":false,"hostName":"qmog.cisco.com","loadBalancer":false}
>>>
>>> {"displayName":"opwv-oam-01","description":"opwv-oam-01
>>> Cartridge","cartridgeAlias":"-","cartridgeType":"opwv-oam-01","activeInstances":0,"provider":"cisco","version":"1","multiTenant":false,"hostName":"qmog.cisco.com","loadBalancer":false}
>>>
>>> {"displayName":"opwv-oam-02","description":"opwv-oam-02
>>> Cartridge","cartridgeAlias":"-","cartridgeType":"opwv-oam-02","activeInstances":0,"provider":"cisco","version":"1","multiTenant":false,"hostName":"qmog.cisco.com","loadBalancer":false}
>>>
>>>
>>>
>>> Deployment policies :
>>>
>>>
>>>
>>>
>>> {"id":"static-1","partitionGroup":[{"id":"N1","partitionAlgo":"one-after-another","partition":[{"id":"RegionOne","partitionMin":1,"partitionMax":1,"provider":"openstack-Core","property":[{"name":"region","value":"RegionOne"}]}]}]}
>>>
>>>
>>> {"id":"autoscale-1-2","partitionGroup":[{"id":"N1","partitionAlgo":"one-after-another","partition":[{"id":"RegionOne","pa
>>>
>>>
>>> rtitionMin":1,"partitionMax":2,"provider":"openstack-Core","property":[{"name":"region","value":"RegionOne"}]}]}]}
>>>
>>>
>>>
>>> Scaling policy :
>>>
>>>
>>>
>>>
>>> {"id":"economyPolicy","loadThresholds":{"requestsInFlight":{"average":300.0,"secondDerivative":0.0,"gradient":0.0,"scaleDownMarginOfGradient":1.0,"scaleDownMarginOfSecondDerivative":0.2},"memoryConsumption":{"average":6000.0,"secondDerivative":0.0,"gradient":0.0,"scaleDownMarginOfGradient":1.0,"scaleDownMarginOfSecondDerivative":0.2},"loadAverage":{"average":40.0,"secondDerivative":0.0,"gradient":0.0,"scaleDownMarginOfGradient":1.0,"scaleDownMarginOfSecondDerivative":0.2}}}
>>>
>>>
>>>
>>> If we subscribe cartridge opwv-vos by itself :
>>>
>>>
>>>
>>> TID: [0] [STRATOS] [2014-07-31 15:15:39,836] INFO
>>> {org.apache.stratos.manager.manager.CartridgeSubscriptionManager} -
>>> Successful Subscription: CartridgeSubscription [subscriptionId=0,
>>> type=opwv-vos, alias=opwv-vos, autoscalingPolicyName=economyPolicy,
>>> deploymentPolicyName=autoscale-1-5, subscriber=Subscriber
>>> [adminUserName=admin, tenantId=-1234, tenantDomain=carbon.super],
>>> repository=Repository [id=0, url=null, userName=,
>>> isPrivateRepository=false],
>>> cartridgeInfo=org.apache.stratos.cloud.controller.stub.pojo.CartridgeInfo@5288e5b6,
>>> payload=SERVICE_NAME=opwv-vos,HOST_NAME=opwv-vos.qmog.cisco.com,MULTITENANT=false,TENANT_ID=-1234,TENANT_RANGE=-1234,CARTRIDGE_ALIAS=opwv-vos,CLUSTER_ID=opwv-vos.opwv-vos.domain,CARTRIDGE_KEY=J5xTyGg9k1od0Dvl,REPO_URL=null,PORTS=22,PROVIDER=cisco,PUPPET_IP=PUPPET_IP,PUPPET_HOSTNAME=PUPPET_HOSTNAME,PUPPET_ENV=PUPPET_ENV,OPWV_INTEGRA_oam_ro=opwv-oam-02,TRUSTSTORE_PASSWORD=wso2carbon,OPWV_INTEGRA_fe_server_type=VOS,OPWV_INTEGRA_wait_for_hosts=oam01~oam02,CEP_PORT=7611,MONITORING_SERVER_SECURE_PORT=0,NO_CARTRIDGE_SUBSCRIBE=false,MB_PORT=61616,MB_IP=octl.qmog.cisco.com,CEP_IP=octl.qmog.cisco.com,DEPLOYMENT=default,OPWV_INTEGRA_region=Core,ENABLE_DATA_PUBLISHER=false,OPWV_INTEGRA_swap_size=2G,MONITORING_SERVER_ADMIN_PASSWORD=xxxx,MONITORING_SERVER_IP=octl.qmog.cisco.com,COMMIT_ENABLED=false,MONITORING_SERVER_ADMIN_USERNAME=xxxx,OPWV_INTEGRA_oam_server_role=,CERT_TRUSTSTORE=/opt/apache-stratos-cartridge-agent/security/client-truststore.jks,OPWV_INTEGRA_oam_server_type=OAMClient,MONITORING_SERVER_PORT=0,OPWV_INTEGRA_oam_rw=opwv-oam-01,OPWV_INTEGRA_sys_component=Core~CC,
>>> cluster=opwv-vos.opwv-vos.domain], subscriptionDomainMap={}
>>> {org.apache.stratos.manager.manager.CartridgeSubscriptionManager}
>>>
>>>
>>>
>>> VM is created, we run a stress load on that VM, we see the load
>>> average increase in aggregator.log, and the autoscaler correctly kicks in
>>> and spawns a second instance :
>>>
>>>
>>>
>>> TID: [0] [STRATOS] [2014-07-31 15:24:42,614] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scale-up] Partition
>>> available, hence trying to spawn an instance to scale up!
>>> {org.apache.stratos.autoscaler.rule.RuleLog}
>>>
>>>
>>>
>>> However when we subscribe all the cartridges together, a stress on
>>> the op-vos instance does not trigger autoscale (although the aggregate log
>>> correctly reports the load) .
>>>
>>>
>>>
>>> Logs are attached.
>>>
>>> Run with single subscription + successful auto scale is from
>>> 2014-07-31 16:03:25,320 -> 2014-07-31 16:07:48,220
>>>
>>> Run with multiple subscription and no auto scale occurring, is from
>>> to 2014-07-31 15:59:50,635 -> 2014-07-31 15:50:08,122
>>>
>>>
>>>
>>> Apologies for hitting the dev list but there doesn’t appear to be a
>>> user list; If anyone can take a look appreciated J
>>>
>>>
>>>
>>>
>>>
>>> Thanks
>>>
>>> David.
>>
>>
>>
>>
>> --
>> Best Regards,
>> Nirmal
>>
>> Nirmal Fernando.
>> PPMC Member & Committer of Apache Stratos,
>> Senior Software Engineer, WSO2 Inc.
>>
>> Blog: http://nirmalfdo.blogspot.com/
>
>
>
>
> --
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
> Blog: http://nirmalfdo.blogspot.com/
--
Akila Ravihansa Perera
Software Engineer
WSO2 Inc.
http://wso2.com
Phone: +94 77 64 154 38
Blog: http://ravihansa3000.blogspot.com
