Hi, Team The DEBUG log has been attached, looks like the issue from LB, do you have any detail instructions for the LB? I created the LB cartridge same as PHP just change the service name to lb, and the lb json as:
{ "type": "lb", "provider": "apache", "displayName": "Lb", "description": "Lb Cartridge", "version": "1.0", "host": "mytesting.cisco.com", "defaultAutoscalingPolicy": "economyPolicy", "multiTenant": "false", "portMapping": [ { "protocol": "http", "port": "22", "proxyPort": "8280" } ], "deployment": {}, "iaasProvider": [ { "type": "openstack", "imageId": "us-internal-1/12345ecf-b91a-4097-929c-b446debe0ee8", "property": [ { "name": "instanceType", "value": "us-internal-1/123453d9-ce0a-472c-944b-3e39476970d3" } ], "networkInterfaces": [ { "networkUuid": "2d84eaa4-1234-4dc8-9897-dd8ef4719f8b" }, { "networkUuid": "2ce8a819-1234-4a07-8248-3c3befe556ac" } ] } ], "property": [ { "name": "load.balancer", "value": "true" } ] } [2015-03-08 19:22:40,634] DEBUG {org.apache.stratos.autoscaler.internal.ASBundleActivater} - AutoScaler bundle is activated. [2015-03-08 19:22:43,159] DEBUG {org.apache.stratos.autoscaler.internal.AutoscalerServerComponent} - Setting the Registry Service [2015-03-08 19:22:43,201] DEBUG {org.apache.stratos.autoscaler.internal.AutoscalerServerComponent} - Topology receiver thread started [2015-03-08 19:22:43,217] DEBUG {org.apache.stratos.autoscaler.internal.AutoscalerServerComponent} - Health message processor thread started [2015-03-08 19:22:43,244] DEBUG {org.apache.stratos.autoscaler.registry.RegistryManager} - Partition read from registry: [id] zone-1 [provider] openstack [min] 0 [max] 0 [2015-03-08 19:22:43,261] DEBUG {org.apache.stratos.autoscaler.registry.RegistryManager} - NetworkPartitionLbHolder read from registry: NetworkPartitionLbHolder [networkPartitionId=openstack-region1, defaultLbClusterId=null, serviceNameToLBClusterIdMap={}, clusterIdToLBClusterIdMap={}] [2015-03-08 19:22:43,276] DEBUG {org.apache.stratos.autoscaler.registry.RegistryManager} - Autoscaler policy read from registry: [id] economyPolicy [name] null [description] null [2015-03-08 19:22:43,281] DEBUG {org.apache.stratos.autoscaler.registry.RegistryManager} - Autoscaler policy read from registry: [id] economyPolicy2 [name] null [description] null [2015-03-08 19:22:43,286] DEBUG {org.apache.stratos.autoscaler.registry.RegistryManager} - Autoscaler policy read from registry: [id] economyPolicy3 [name] null [description] null [2015-03-08 19:22:43,291] DEBUG {org.apache.stratos.autoscaler.registry.RegistryManager} - Autoscaler policy read from registry: [id] economyPolicy4 [name] null [description] null [2015-03-08 19:22:43,293] DEBUG {org.apache.stratos.autoscaler.policy.PolicyManager} - Adding policy :economyPolicy [2015-03-08 19:22:43,293] DEBUG {org.apache.stratos.autoscaler.policy.PolicyManager} - Adding policy :economyPolicy2 [2015-03-08 19:22:43,293] DEBUG {org.apache.stratos.autoscaler.policy.PolicyManager} - Adding policy :economyPolicy3 [2015-03-08 19:22:43,293] DEBUG {org.apache.stratos.autoscaler.policy.PolicyManager} - Adding policy :economyPolicy4 [2015-03-08 19:22:43,305] DEBUG {org.apache.stratos.autoscaler.registry.RegistryManager} - Deployment Policy [id]economyDeployment [partitions] [org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@6c2c08ed ] [2015-03-08 19:22:43,310] DEBUG {org.apache.stratos.autoscaler.registry.RegistryManager} - Deployment Policy [id]economyDeployment2 [partitions] [org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@65e19150 ] [2015-03-08 19:22:43,314] DEBUG {org.apache.stratos.autoscaler.registry.RegistryManager} - Deployment Policy [id]economyDeployment201 [partitions] [org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@29d4fae8 ] [2015-03-08 19:22:43,319] DEBUG {org.apache.stratos.autoscaler.registry.RegistryManager} - Deployment Policy [id]economyDeployment3 [partitions] [org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@18711ad ] [2015-03-08 19:22:43,319] DEBUG {org.apache.stratos.autoscaler.policy.PolicyManager} - Adding policy :economyDeployment [2015-03-08 19:22:43,319] DEBUG {org.apache.stratos.autoscaler.policy.PolicyManager} - Adding policy :economyDeployment2 [2015-03-08 19:22:43,319] DEBUG {org.apache.stratos.autoscaler.policy.PolicyManager} - Adding policy :economyDeployment201 [2015-03-08 19:22:43,319] DEBUG {org.apache.stratos.autoscaler.policy.PolicyManager} - Adding policy :economyDeployment3 [2015-03-08 19:22:43,319] INFO {org.apache.stratos.autoscaler.internal.AutoscalerServerComponent} - Autoscaler Server Component activated [2015-03-08 19:22:54,871] INFO {org.springframework.beans.factory.xml.XmlBeanDefinitionReader} - Loading XML bean definitions from class path resource [META-INF/cxf/cxf.xml] [2015-03-08 19:22:54,966] INFO {org.springframework.beans.factory.xml.XmlBeanDefinitionReader} - Loading XML bean definitions from URL [jndi:/localhost/stratos/WEB-INF/cxf-servlet.xml] [2015-03-08 19:22:56,147] INFO {org.wso2.carbon.core.deployment.DeploymentInterceptor} - Deploying Axis2 service: AutoScalerService {super-tenant} [2015-03-08 19:22:56,296] INFO {org.apache.axis2.deployment.DeploymentEngine} - Deploying Web service: org.apache.stratos.autoscaler - [2015-03-08 19:22:58,209] INFO {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver} - Autoscaler topology receiver thread started [2015-03-08 19:22:58,217] INFO {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver} - Autoscaler health stat event receiver thread started [2015-03-08 19:27:19,829] INFO {org.apache.stratos.rest.endpoint.services.ServiceUtils} - Starting to deploy a Cartridge [type] lb [2015-03-08 19:27:22,117] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - Successfully undeployed the Cartridge definition: lb [2015-03-08 19:27:22,148] INFO {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - Publishing service created event: [service] lb [2015-03-08 19:27:22,152] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - Successfully deployed the Cartridge definition: lb [2015-03-08 19:27:22,152] WARN {org.apache.stratos.messaging.message.processor.topology.ServiceCreatedMessageProcessor} - Service already created: [service] lb [2015-03-08 19:27:22,153] WARN {org.apache.stratos.messaging.message.processor.topology.ServiceCreatedMessageProcessor} - Service already created: [service] lb [2015-03-08 19:27:22,154] INFO {org.apache.stratos.rest.endpoint.services.ServiceUtils} - Successfully deployed Cartridge [type] lb [2015-03-08 19:27:31,313] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - Successfully undeployed the Cartridge definition: lb [2015-03-08 19:27:41,358] INFO {org.apache.stratos.rest.endpoint.services.ServiceUtils} - Starting to deploy a Cartridge [type] lb [2015-03-08 19:27:42,473] INFO {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - Publishing service created event: [service] lb [2015-03-08 19:27:42,478] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - Successfully deployed the Cartridge definition: lb [2015-03-08 19:27:42,478] WARN {org.apache.stratos.messaging.message.processor.topology.ServiceCreatedMessageProcessor} - Service already created: [service] lb [2015-03-08 19:27:42,479] WARN {org.apache.stratos.messaging.message.processor.topology.ServiceCreatedMessageProcessor} - Service already created: [service] lb [2015-03-08 19:27:42,479] INFO {org.apache.stratos.rest.endpoint.services.ServiceUtils} - Successfully deployed Cartridge [type] lb [2015-03-08 19:28:16,709] INFO {org.apache.stratos.autoscaler.util.ConfUtil} - Loading configuration..... [2015-03-08 19:28:16,759] INFO {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Validating partitions of policy via cloud controller: [id] economyDeployment201 [2015-03-08 19:28:18,065] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - All partitions [zone-1] were validated successfully, against the Cartridge: lb [2015-03-08 19:28:18,070] DEBUG {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Service call validateDeploymentPolicy() returned in 1310ms [2015-03-08 19:28:18,071] INFO {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Validating partitions of policy via cloud controller: [id] economyDeployment3 [2015-03-08 19:28:19,146] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - All partitions [zone-1] were validated successfully, against the Cartridge: lb [2015-03-08 19:28:19,149] DEBUG {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Service call validateDeploymentPolicy() returned in 1078ms [2015-03-08 19:28:19,149] INFO {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Validating partitions of policy via cloud controller: [id] economyDeployment2 [2015-03-08 19:28:20,160] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - All partitions [zone-1] were validated successfully, against the Cartridge: lb [2015-03-08 19:28:20,163] DEBUG {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Service call validateDeploymentPolicy() returned in 1014ms [2015-03-08 19:28:20,164] INFO {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Validating partitions of policy via cloud controller: [id] economyDeployment [2015-03-08 19:28:21,170] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - All partitions [zone-1] were validated successfully, against the Cartridge: lb [2015-03-08 19:28:21,173] DEBUG {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Service call validateDeploymentPolicy() returned in 1009ms [2015-03-08 19:28:39,689] INFO {org.apache.stratos.manager.manager.CartridgeSubscriptionManager} - Tenant [-1234] with username [ADMIN subscribed to ] Cartridge with Alias lb, Cartridge Type: lb, Repo URL: https://github.com/prodstagegit/ieptestb.git, Autoscale Policy: economyPolicy, Deployment Policy: economyDeployment [2015-03-08 19:28:39,689] INFO {org.apache.stratos.manager.behaviour.CartridgeMgtBehaviour} - Payload: SERVICE_NAME=lb,HOST_NAME=lb.mytesting.prodstage.com ,MULTITENANT=false,TENANT_ID=-1234,TENANT_RANGE=-1234,CARTRIDGE_ALIAS=lb,CLUSTER_ID=lb.lb.domain,CARTRIDGE_KEY=pRXTFJIUj64f1yNs,DEPLOYMENT=default,REPO_URL= https://github.com/prodstagegit/ieptestb.git,PORTS=22,PUPPET_IP=127.0.0.1,PUPPET_HOSTNAME=stratos40build05.prodstage.com,PUPPET_ENV=puppet_environment,COMMIT_ENABLED=false [2015-03-08 19:28:39,722] INFO {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - Publishing cluster created event: Cluster [serviceName=lb, clusterId=lb.lb.domain, autoscalePolicyName=economyPolicy, deploymentPolicyName=economyDeployment, hostNames=[ lb.mytesting.prodstage.com], tenantRange=null, isLbCluster=false, properties={}] [2015-03-08 19:28:39,729] INFO {org.apache.stratos.messaging.message.processor.topology.ClusterCreatedMessageProcessor} - Cluster created: Cluster [serviceName=lb, clusterId=lb.lb.domain, autoscalePolicyName=economyPolicy, deploymentPolicyName=economyDeployment, hostNames=[lb.mytesting.prodstage.com], tenantRange=null, isLbCluster=false, properties={}] [2015-03-08 19:28:39,731] WARN {org.apache.stratos.messaging.message.processor.topology.ClusterCreatedMessageProcessor} - Cluster already exists in service: [service] lb [cluster] lb.lb.domain [2015-03-08 19:28:39,732] INFO {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver} - Event received: ClusterCreatedEvent [serviceName=lb, cluster=Cluster [serviceName=lb, clusterId=lb.lb.domain, autoscalePolicyName=economyPolicy, deploymentPolicyName=economyDeployment, hostNames=[ lb.mytesting.prodstage.com], tenantRange=null, isLbCluster=false, properties={}]] [2015-03-08 19:28:39,758] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - Successfully registered: Registrant [clusterId=lb.lb.domain, tenantRange=-1234, hostName=lb.mytesting.prodstage.com, cartridgeType=lb, properties=Properties [properties=null], autoScalerPolicyName=economyPolicy, deploymentPolicyName=economyDeployment] [2015-03-08 19:28:39,802] INFO {org.apache.stratos.manager.manager.CartridgeSubscriptionManager} - Successful Subscription: CartridgeSubscription [subscriptionId=0, type=lb, alias=lb, autoscalingPolicyName=economyPolicy, deploymentPolicyName=economyDeployment, subscriber=Subscriber [adminUserName=ADMIN, tenantId=-1234, tenantDomain=carbon.super], repository=Repository [id=0, url= https://github.com/prodstagegit/ieptestb.git, userName=null, isPrivateRepository=false], cartridgeInfo=org.apache.stratos.cloud.controller.stub.pojo.CartridgeInfo@2f0e2af2, payload=SERVICE_NAME=lb,HOST_NAME=lb.mytesting.prodstage.com ,MULTITENANT=false,TENANT_ID=-1234,TENANT_RANGE=-1234,CARTRIDGE_ALIAS=lb,CLUSTER_ID=lb.lb.domain,CARTRIDGE_KEY=pRXTFJIUj64f1yNs,DEPLOYMENT=default,REPO_URL= https://github.com/prodstagegit/ieptestb.git,PORTS=22,PUPPET_IP=127.0.0.1,PUPPET_HOSTNAME=stratos40build05.prodstage.com,PUPPET_ENV=puppet_environment,COMMIT_ENABLED=false, cluster=lb.lb.domain] [2015-03-08 19:28:39,804] INFO {org.apache.stratos.manager.subscription.utils.CartridgeSubscriptionUtils} - Publishing tenant subscribed event: [tenant-id] -1234 [service] lb [2015-03-08 19:28:44,736] DEBUG {org.apache.stratos.autoscaler.util.AutoscalerUtil} - Deployment policy name: economyDeployment [2015-03-08 19:28:44,737] DEBUG {org.apache.stratos.autoscaler.util.AutoscalerUtil} - Autoscaler policy name: economyPolicy [2015-03-08 19:28:44,737] INFO {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Validating partitions of policy via cloud controller: [id] economyDeployment [2015-03-08 19:28:45,732] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - All partitions [zone-1] were validated successfully, against the Cartridge: lb [2015-03-08 19:28:45,735] DEBUG {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Service call validateDeploymentPolicy() returned in 997ms [2015-03-08 19:28:46,293] DEBUG {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum check rule is parsed successfully [2015-03-08 19:28:46,599] DEBUG {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Scale check rule is parsed successfully [2015-03-08 19:28:46,629] DEBUG {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Terminate all rule is parsed successfully [2015-03-08 19:28:46,669] DEBUG {org.apache.stratos.autoscaler.PartitionContext} - Member expiry time is set to: 900000 [2015-03-08 19:28:46,670] INFO {org.apache.stratos.autoscaler.util.AutoscalerUtil} - Partition context has been added: [partition] zone-1 [2015-03-08 19:28:46,671] INFO {org.apache.stratos.autoscaler.util.AutoscalerUtil} - Network partition context has been added: [network partition] openstack-region1 [2015-03-08 19:28:46,671] INFO {org.apache.stratos.autoscaler.util.AutoscalerUtil} - Cluster monitor created: ClusterMonitor [clusterId=lb.lb.domain, serviceId=lb, deploymentPolicy=Deployment Policy [id]economyDeployment [partitions] [org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@6c2c08ed], autoscalePolicy=ASPolicy [id=economyPolicy, displayName=null, description=null], lbReferenceType=null] [2015-03-08 19:28:46,671] INFO {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver} - Cluster monitor has been added successfully: [cluster] lb.lb.domain [2015-03-08 19:28:46,672] DEBUG {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver} - Cluster monitor thread has been started successfully: [cluster] lb.lb.domain [2015-03-08 19:28:50,074] INFO {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Validating partitions of policy via cloud controller: [id] economyDeployment201 [2015-03-08 19:28:51,745] DEBUG {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Service call validateDeploymentPolicy() returned in 1670ms [2015-03-08 19:28:51,745] INFO {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Validating partitions of policy via cloud controller: [id] economyDeployment3 [2015-03-08 19:28:52,639] DEBUG {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Service call validateDeploymentPolicy() returned in 894ms [2015-03-08 19:28:52,639] INFO {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Validating partitions of policy via cloud controller: [id] economyDeployment2 [2015-03-08 19:28:53,537] DEBUG {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Service call validateDeploymentPolicy() returned in 897ms [2015-03-08 19:28:53,537] INFO {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Validating partitions of policy via cloud controller: [id] economyDeployment [2015-03-08 19:28:54,454] DEBUG {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Service call validateDeploymentPolicy() returned in 917ms [2015-03-08 19:29:10,105] INFO {org.apache.stratos.manager.manager.CartridgeSubscriptionManager} - Tenant [-1234] with username [ADMIN subscribed to ] Cartridge with Alias php, Cartridge Type: php301, Repo URL: https://github.com/prodstagegit/hbase.git, Autoscale Policy: economyPolicy4, Deployment Policy: economyDeployment [2015-03-08 19:29:10,147] INFO {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - Publishing cluster created event: Cluster [serviceName=php301, clusterId=php.php301.domain, autoscalePolicyName=economyPolicy4, deploymentPolicyName=economyDeployment, hostNames=[php.cartridge-php], tenantRange=null, isLbCluster=false, properties={}] [2015-03-08 19:29:10,154] INFO {org.apache.stratos.messaging.message.processor.topology.ClusterCreatedMessageProcessor} - Cluster created: Cluster [serviceName=php301, clusterId=php.php301.domain, autoscalePolicyName=economyPolicy4, deploymentPolicyName=economyDeployment, hostNames=[php.cartridge-php], tenantRange=null, isLbCluster=false, properties={}] [2015-03-08 19:29:10,154] INFO {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver} - Event received: ClusterCreatedEvent [serviceName=php301, cluster=Cluster [serviceName=php301, clusterId=php.php301.domain, autoscalePolicyName=economyPolicy4, deploymentPolicyName=economyDeployment, hostNames=[php.cartridge-php], tenantRange=null, isLbCluster=false, properties={}]] [2015-03-08 19:29:10,178] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - Successfully registered: Registrant [clusterId=php.php301.domain, tenantRange=-1234, hostName=php.cartridge-php, cartridgeType=php301, properties=Properties [properties=null], autoScalerPolicyName=economyPolicy4, deploymentPolicyName=economyDeployment] [2015-03-08 19:29:10,207] INFO {org.apache.stratos.manager.manager.CartridgeSubscriptionManager} - Successful Subscription: CartridgeSubscription [subscriptionId=0, type=php301, alias=php, autoscalingPolicyName=economyPolicy4, deploymentPolicyName=economyDeployment, subscriber=Subscriber [adminUserName=ADMIN, tenantId=-1234, tenantDomain=carbon.super], repository=Repository [id=0, url=https://github.com/prodstagegit/hbase.git, userName=null, isPrivateRepository=false], cartridgeInfo=org.apache.stratos.cloud.controller.stub.pojo.CartridgeInfo@5cc4e0d6, payload=SERVICE_NAME=php301,HOST_NAME=php.cartridge-php,MULTITENANT=false,TENANT_ID=-1234,TENANT_RANGE=-1234,CARTRIDGE_ALIAS=php,CLUSTER_ID=php.php301.domain,CARTRIDGE_KEY=LRNIQbYRheV61YmX,DEPLOYMENT=default,REPO_URL= https://github.com/prodstagegit/hbase.git,PORTS=80,PUPPET_IP=127.0.0.1,PUPPET_HOSTNAME=stratos40build05.prodstage.com,PUPPET_ENV=puppet_environment,COMMIT_ENABLED=false, cluster=php.php301.domain] [2015-03-08 19:29:15,155] DEBUG {org.apache.stratos.autoscaler.util.AutoscalerUtil} - Deployment policy name: economyDeployment [2015-03-08 19:29:15,157] DEBUG {org.apache.stratos.autoscaler.util.AutoscalerUtil} - Autoscaler policy name: economyPolicy4 [2015-03-08 19:29:15,157] INFO {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Validating partitions of policy via cloud controller: [id] economyDeployment [2015-03-08 19:29:15,948] DEBUG {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Service call validateDeploymentPolicy() returned in 791ms [2015-03-08 19:29:15,989] DEBUG {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum check rule is parsed successfully [2015-03-08 19:29:16,206] DEBUG {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Scale check rule is parsed successfully [2015-03-08 19:29:16,233] DEBUG {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Terminate all rule is parsed successfully [2015-03-08 19:29:16,234] DEBUG {org.apache.stratos.autoscaler.PartitionContext} - Member expiry time is set to: 900000 [2015-03-08 19:29:16,234] INFO {org.apache.stratos.autoscaler.util.AutoscalerUtil} - Partition context has been added: [partition] zone-1 [2015-03-08 19:29:16,234] INFO {org.apache.stratos.autoscaler.util.AutoscalerUtil} - Network partition context has been added: [network partition] openstack-region1 [2015-03-08 19:29:16,235] INFO {org.apache.stratos.autoscaler.util.AutoscalerUtil} - Cluster monitor created: ClusterMonitor [clusterId=php.php301.domain, serviceId=php301, deploymentPolicy=Deployment Policy [id]economyDeployment [partitions] [org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@6c2c08ed], autoscalePolicy=ASPolicy [id=economyPolicy4, displayName=null, description=null], lbReferenceType=null] [2015-03-08 19:29:16,235] INFO {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver} - Cluster monitor has been added successfully: [cluster] php.php301.domain [2015-03-08 19:29:16,235] DEBUG {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver} - Cluster monitor thread has been started successfully: [cluster] php.php301.domain [2015-03-08 19:29:46,672] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Cluster monitor is running.. ClusterMonitor [clusterId=lb.lb.domain, serviceId=lb, deploymentPolicy=Deployment Policy [id]economyDeployment [partitions] [org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@6c2c08ed], autoscalePolicy=ASPolicy [id=economyPolicy, displayName=null, description=null], lbReferenceType=null] [2015-03-08 19:29:46,672] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Running minimum check for partition zone-1 [2015-03-08 19:29:46,687] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule: [network-partition] openstack-region1 [partition] zone-1 [2015-03-08 19:29:46,688] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [network-partition] openstack-region1 [partition] zone-1 Non terminated member count: 0 [2015-03-08 19:29:46,688] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [network-partition] openstack-region1 [partition] zone-1 Minimum member count: 1 [2015-03-08 19:29:46,690] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete instances rule: [partition] zone-1 [network-partition] openstack-region1 [2015-03-08 19:29:46,691] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [network-partition] openstack-region1 [partition] zone-1 Obsolete member count: 0 [2015-03-08 19:29:46,692] DEBUG {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Getting LB id for spawning instance [lb reference] null , [partition] zone-1 [network partition] openstack-region1 [Lb id] null [2015-03-08 19:29:46,692] INFO {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Trying to spawn an instance via cloud controller: [cluster] lb.lb.domain [partition] zone-1 [lb-cluster] null [network-partition-id] openstack-region1 [2015-03-08 19:30:00,106] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - Instance is successfully starting up. MemberContext [memberId=lb.lb.domain2604c1c5-a28c-4b79-8ce8-731e6605d21f, nodeId=us-internal-1/4ed8a54c-f9d6-42ed-99a7-216619354d42, clusterId=lb.lb.domain, cartridgeType=lb, privateIpAddress=null, publicIpAddress=null, allocatedIpAddress=null, initTime=1425842986692, lbClusterId=null, networkPartitionId=openstack-region1] [2015-03-08 19:30:00,106] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - Retrieving Private IP Address. MemberContext [memberId=lb.lb.domain2604c1c5-a28c-4b79-8ce8-731e6605d21f, nodeId=us-internal-1/4ed8a54c-f9d6-42ed-99a7-216619354d42, clusterId=lb.lb.domain, cartridgeType=lb, privateIpAddress=192.168.21.121, publicIpAddress=null, allocatedIpAddress=null, initTime=1425842986692, lbClusterId=null, networkPartitionId=openstack-region1] [2015-03-08 19:30:00,115] DEBUG {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Service call startInstance() returned in 13423ms [2015-03-08 19:30:00,117] DEBUG {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Pending member added, [member] lb.lb.domain2604c1c5-a28c-4b79-8ce8-731e6605d21f [partition] zone-1 [2015-03-08 19:30:00,118] DEBUG {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum check executed for : org.apache.stratos.autoscaler.PartitionContext@65cd1989 [2015-03-08 19:30:00,118] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Scale rule will not run since the LB statistics have not received before this cycle for network partition openstack-region1 [2015-03-08 19:30:00,158] INFO {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - Publishing instance spawned event: [service] lb [cluster] lb.lb.domain [network-partition] openstack-region1 [partition] zone-1 [member] lb.lb.domain2604c1c5-a28c-4b79-8ce8-731e6605d21f [lb-cluster-id] null [2015-03-08 19:30:00,162] INFO {org.apache.stratos.messaging.message.processor.topology.InstanceSpawnedMessageProcessor} - Member created: [service] lb [cluster] lb.lb.domain [member] lb.lb.domain2604c1c5-a28c-4b79-8ce8-731e6605d21f [2015-03-08 19:30:00,163] WARN {org.apache.stratos.messaging.message.processor.topology.InstanceSpawnedMessageProcessor} - Member already exists: [service] lb [cluster] lb.lb.domain [member] lb.lb.domain2604c1c5-a28c-4b79-8ce8-731e6605d21f [2015-03-08 19:30:16,235] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Cluster monitor is running.. ClusterMonitor [clusterId=php.php301.domain, serviceId=php301, deploymentPolicy=Deployment Policy [id]economyDeployment [partitions] [org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@6c2c08ed], autoscalePolicy=ASPolicy [id=economyPolicy4, displayName=null, description=null], lbReferenceType=null] [2015-03-08 19:30:16,237] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Running minimum check for partition zone-1 [2015-03-08 19:30:16,238] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule: [network-partition] openstack-region1 [partition] zone-1 [2015-03-08 19:30:16,239] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [network-partition] openstack-region1 [partition] zone-1 Non terminated member count: 0 [2015-03-08 19:30:16,239] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [network-partition] openstack-region1 [partition] zone-1 Minimum member count: 1 [2015-03-08 19:30:16,240] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete instances rule: [partition] zone-1 [network-partition] openstack-region1 [2015-03-08 19:30:16,240] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [network-partition] openstack-region1 [partition] zone-1 Obsolete member count: 0 [2015-03-08 19:30:16,241] DEBUG {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Getting LB id for spawning instance [lb reference] null , [partition] zone-1 [network partition] openstack-region1 [Lb id] null [2015-03-08 19:30:16,241] INFO {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Trying to spawn an instance via cloud controller: [cluster] php.php301.domain [partition] zone-1 [lb-cluster] null [network-partition-id] openstack-region1 [2015-03-08 19:30:30,118] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Cluster monitor is running.. ClusterMonitor [clusterId=lb.lb.domain, serviceId=lb, deploymentPolicy=Deployment Policy [id]economyDeployment [partitions] [org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@6c2c08ed], autoscalePolicy=ASPolicy [id=economyPolicy, displayName=null, description=null], lbReferenceType=null] [2015-03-08 19:30:30,119] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Running minimum check for partition zone-1 [2015-03-08 19:30:30,121] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule: [network-partition] openstack-region1 [partition] zone-1 [2015-03-08 19:30:30,121] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [network-partition] openstack-region1 [partition] zone-1 Non terminated member count: 1 [2015-03-08 19:30:30,121] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [network-partition] openstack-region1 [partition] zone-1 Minimum member count: 1 [2015-03-08 19:30:30,123] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete instances rule: [partition] zone-1 [network-partition] openstack-region1 [2015-03-08 19:30:30,123] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [network-partition] openstack-region1 [partition] zone-1 Obsolete member count: 0 [2015-03-08 19:30:30,123] DEBUG {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum check executed for : org.apache.stratos.autoscaler.PartitionContext@65cd1989 [2015-03-08 19:30:30,123] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Scale rule will not run since the LB statistics have not received before this cycle for network partition openstack-region1 [2015-03-08 19:30:30,456] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - Instance is successfully starting up. MemberContext [memberId=php.php301.domain2da77371-b137-4bd0-a4f0-dbc2f8ad0843, nodeId=us-internal-1/bafb885c-fd0d-4893-a58f-e58082fdc245, clusterId=php.php301.domain, cartridgeType=php301, privateIpAddress=null, publicIpAddress=null, allocatedIpAddress=null, initTime=1425843016242, lbClusterId=null, networkPartitionId=openstack-region1] [2015-03-08 19:30:30,456] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - Retrieving Private IP Address. MemberContext [memberId=php.php301.domain2da77371-b137-4bd0-a4f0-dbc2f8ad0843, nodeId=us-internal-1/bafb885c-fd0d-4893-a58f-e58082fdc245, clusterId=php.php301.domain, cartridgeType=php301, privateIpAddress=192.168.21.122, publicIpAddress=null, allocatedIpAddress=null, initTime=1425843016242, lbClusterId=null, networkPartitionId=openstack-region1] [2015-03-08 19:30:30,465] DEBUG {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - Service call startInstance() returned in 14223ms [2015-03-08 19:30:30,466] DEBUG {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Pending member added, [member] php.php301.domain2da77371-b137-4bd0-a4f0-dbc2f8ad0843 [partition] zone-1 [2015-03-08 19:30:30,466] DEBUG {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum check executed for : org.apache.stratos.autoscaler.PartitionContext@695c1d4e [2015-03-08 19:30:30,466] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Scale rule will not run since the LB statistics have not received before this cycle for network partition openstack-region1 [2015-03-08 19:30:30,512] INFO {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - Publishing instance spawned event: [service] php301 [cluster] php.php301.domain [network-partition] openstack-region1 [partition] zone-1 [member] php.php301.domain2da77371-b137-4bd0-a4f0-dbc2f8ad0843 [lb-cluster-id] null [2015-03-08 19:31:00,124] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Cluster monitor is running.. ClusterMonitor [clusterId=lb.lb.domain, serviceId=lb, deploymentPolicy=Deployment Policy [id]economyDeployment [partitions] [org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@6c2c08ed], autoscalePolicy=ASPolicy [id=economyPolicy, displayName=null, description=null], lbReferenceType=null] [2015-03-08 19:31:00,124] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Running minimum check for partition zone-1 [2015-03-08 19:31:00,126] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule: [network-partition] openstack-region1 [partition] zone-1 [2015-03-08 19:31:00,126] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [network-partition] openstack-region1 [partition] zone-1 Non terminated member count: 1 [2015-03-08 19:31:00,126] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [network-partition] openstack-region1 [partition] zone-1 Minimum member count: 1 [2015-03-08 19:31:00,126] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete instances rule: [partition] zone-1 [network-partition] openstack-region1 [2015-03-08 19:31:00,126] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [network-partition] openstack-region1 [partition] zone-1 Obsolete member count: 0 [2015-03-08 19:31:00,126] DEBUG {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum check executed for : org.apache.stratos.autoscaler.PartitionContext@65cd1989 [2015-03-08 19:31:00,127] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Scale rule will not run since the LB statistics have not received before this cycle for network partition openstack-region1 [2015-03-08 19:31:00,466] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Cluster monitor is running.. ClusterMonitor [clusterId=php.php301.domain, serviceId=php301, deploymentPolicy=Deployment Policy [id]economyDeployment [partitions] [org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@6c2c08ed], autoscalePolicy=ASPolicy [id=economyPolicy4, displayName=null, description=null], lbReferenceType=null] [2015-03-08 19:31:00,467] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Running minimum check for partition zone-1 [2015-03-08 19:31:00,467] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule: [network-partition] openstack-region1 [partition] zone-1 [2015-03-08 19:31:00,467] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [network-partition] openstack-region1 [partition] zone-1 Non terminated member count: 1 [2015-03-08 19:31:00,467] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [network-partition] openstack-region1 [partition] zone-1 Minimum member count: 1 [2015-03-08 19:31:00,468] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete instances rule: [partition] zone-1 [network-partition] openstack-region1 [2015-03-08 19:31:00,468] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [network-partition] openstack-region1 [partition] zone-1 Obsolete member count: 0 [2015-03-08 19:31:00,468] DEBUG {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum check executed for : org.apache.stratos.autoscaler.PartitionContext@695c1d4e [2015-03-08 19:31:00,468] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Scale rule will not run since the LB statistics have not received before this cycle for network partition openstack-region1 [2015-03-08 19:31:30,127] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Cluster monitor is running.. ClusterMonitor [clusterId=lb.lb.domain, serviceId=lb, deploymentPolicy=Deployment Policy [id]economyDeployment [partitions] [org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@6c2c08ed], autoscalePolicy=ASPolicy [id=economyPolicy, displayName=null, description=null], lbReferenceType=null] [2015-03-08 19:31:30,127] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Running minimum check for partition zone-1 [2015-03-08 19:31:30,128] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule: [network-partition] openstack-region1 [partition] zone-1 [2015-03-08 19:31:30,128] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [network-partition] openstack-region1 [partition] zone-1 Non terminated member count: 1 [2015-03-08 19:31:30,128] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [network-partition] openstack-region1 [partition] zone-1 Minimum member count: 1 [2015-03-08 19:31:30,128] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete instances rule: [partition] zone-1 [network-partition] openstack-region1 [2015-03-08 19:31:30,128] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [network-partition] openstack-region1 [partition] zone-1 Obsolete member count: 0 [2015-03-08 19:31:30,129] DEBUG {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum check executed for : org.apache.stratos.autoscaler.PartitionContext@65cd1989 [2015-03-08 19:31:30,129] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Scale rule will not run since the LB statistics have not received before this cycle for network partition openstack-region1 [2015-03-08 19:31:30,468] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Cluster monitor is running.. ClusterMonitor [clusterId=php.php301.domain, serviceId=php301, deploymentPolicy=Deployment Policy [id]economyDeployment [partitions] [org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@6c2c08ed], autoscalePolicy=ASPolicy [id=economyPolicy4, displayName=null, description=null], lbReferenceType=null] [2015-03-08 19:31:30,469] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Running minimum check for partition zone-1 [2015-03-08 19:31:30,469] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule: [network-partition] openstack-region1 [partition] zone-1 [2015-03-08 19:31:30,469] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [network-partition] openstack-region1 [partition] zone-1 Non terminated member count: 1 [2015-03-08 19:31:30,469] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [network-partition] openstack-region1 [partition] zone-1 Minimum member count: 1 [2015-03-08 19:31:30,469] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete instances rule: [partition] zone-1 [network-partition] openstack-region1 [2015-03-08 19:31:30,470] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [network-partition] openstack-region1 [partition] zone-1 Obsolete member count: 0 [2015-03-08 19:31:30,470] DEBUG {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum check executed for : org.apache.stratos.autoscaler.PartitionContext@695c1d4e [2015-03-08 19:31:30,470] DEBUG {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Scale rule will not run since the LB statistics have not received before this cycle for network partition openstack-region1 Please help, Thanks -D On Sun, Mar 8, 2015 at 1:06 AM, Rajkumar Rajaratnam <rajkum...@wso2.com> wrote: > Hi David, > > It is not puppet agent which sends health stats to CEP. It is cartridge > agent's responsibility. If you do the following things, we can easily sort > out your issue. > > - enable debug log for following loggers in > (stratos-home/repository/conf//var/log/*apache*-*stratos*/*cartridge*- > *agent*.*log) *file > *log4j.logger.org.apache.stratos.autoscaler=DEBUG* > > * log4j.logger.org.apache.stratos.autoscaler.rule.RuleLog=DEBUG * > - restart the stratos > - send us the wso2carbon.log > (stratos-home/repository/logs/wso2carbon.log) > > Thanks. > > > On Sun, Mar 8, 2015 at 1:58 PM, david hbase <davidhb...@gmail.com> wrote: > >> +Imesh with this emaill chain >> >> The AC ( auto scaling ) still not work. Looks like the puppet agent do >> not send back the system/health check status. >> puppet agent test is good but not health status back to master. >> >> testing result: >> root@phpphp301-7c5:~# puppet agent --noop --test >> Info: Retrieving pluginfacts >> Info: Retrieving plugin >> Info: Loading facts >> Info: Caching catalog for phpphp301-7c5.prodstage.com >> Info: Applying configuration version '1425798573' >> Notice: /Stage[main]/Java/Exec[Install java]/returns: current_value >> notrun, should be 0 (noop) >> Notice: Class[Java]: Would have triggered 'refresh' from 1 events >> Notice: /Stage[main]/Php/Exec[update-apt]/returns: current_value notrun, >> should be 0 (noop) >> Notice: /Stage[main]/Php/Exec[enable ssl module]/returns: current_value >> notrun, should be 0 (noop) >> Notice: /Stage[main]/Php/Exec[enable ssl]/returns: current_value notrun, >> should be 0 (noop) >> Notice: /Stage[main]/Php/Exec[apache2 restart]/returns: current_value >> notrun, should be 0 (noop) >> Notice: Class[Php]: Would have triggered 'refresh' from 4 events >> Info: Class[Php]: Scheduling refresh of Class[Agent] >> Notice: Class[Agent]: Would have triggered 'refresh' from 1 events >> Info: Class[Agent]: Scheduling refresh of >> Agent::Push_templates[extensions/artifacts-updated.sh] >> Info: Class[Agent]: Scheduling refresh of >> Agent::Push_templates[extensions/start-servers.sh] >> Info: Class[Agent]: Scheduling refresh of >> Agent::Push_templates[extensions/instance-activated.sh] >> Info: Class[Agent]: Scheduling refresh of >> Agent::Push_templates[extensions/instance-started.sh] >> Info: Class[Agent]: Scheduling refresh of Agent::Start[cartridge-agent] >> Info: Class[Agent]: Scheduling refresh of >> Agent::Initialize[cartridge-agent] >> Info: Class[Agent]: Scheduling refresh of >> Agent::Push_templates[conf/templates/jndi.properties.template] >> Info: Class[Agent]: Scheduling refresh of >> Agent::Push_templates[bin/stratos.sh] >> Info: Class[Agent]: Scheduling refresh of Exec[copy launch-params to >> carbon_home] >> Info: Class[Agent]: Scheduling refresh of >> Agent::Push_templates[extensions/clean.sh] >> Notice: Agent::Initialize[cartridge-agent]: Would have triggered >> 'refresh' from 1 events >> Info: Agent::Initialize[cartridge-agent]: Scheduling refresh of >> Exec[creating_target_for_cartridge-agent] >> Info: Agent::Initialize[cartridge-agent]: Scheduling refresh of >> Exec[extracting_stratoscartridge-agent-4.0.0.zip_for_cartridge-agent] >> Info: Agent::Initialize[cartridge-agent]: Scheduling refresh of >> Exec[downloading_apache-stratos-cartridge-agent-4.0.0.zip] >> Info: Agent::Initialize[cartridge-agent]: Scheduling refresh of >> Exec[setting_permission_for_cartridge-agent] >> Info: Agent::Initialize[cartridge-agent]: Scheduling refresh of >> Exec[creating_local_package_repo_for_cartridge-agent] >> Notice: >> /Stage[main]/Agent/Agent::Initialize[cartridge-agent]/Exec[creating_local_package_repo_for_cartridge-agent]: >> Would have triggered 'refresh' from 1 events >> Notice: >> /Stage[main]/Agent/Agent::Initialize[cartridge-agent]/Exec[creating_target_for_cartridge-agent]/returns: >> current_value notrun, should be 0 (noop) >> Notice: >> /Stage[main]/Agent/Agent::Initialize[cartridge-agent]/Exec[creating_target_for_cartridge-agent]: >> Would have triggered 'refresh' from 1 events >> Notice: >> /Stage[main]/Agent/Agent::Initialize[cartridge-agent]/Exec[downloading_apache-stratos-cartridge-agent-4.0.0.zip]: >> Would have triggered 'refresh' from 1 events >> Notice: >> /Stage[main]/Agent/Agent::Initialize[cartridge-agent]/Exec[extracting_stratoscartridge-agent-4.0.0.zip_for_cartridge-agent]: >> Would have triggered 'refresh' from 1 events >> Notice: >> /Stage[main]/Agent/Agent::Initialize[cartridge-agent]/Exec[setting_permission_for_cartridge-agent]/returns: >> current_value notrun, should be 0 (noop) >> Notice: >> /Stage[main]/Agent/Agent::Initialize[cartridge-agent]/Exec[setting_permission_for_cartridge-agent]: >> Would have triggered 'refresh' from 1 events >> Notice: Agent::Initialize[cartridge-agent]: Would have triggered >> 'refresh' from 7 events >> Notice: Agent::Push_templates[bin/stratos.sh]: Would have triggered >> 'refresh' from 1 events >> Notice: Agent::Push_templates[extensions/instance-activated.sh]: Would >> have triggered 'refresh' from 1 events >> Notice: /Stage[main]/Agent/Exec[copy launch-params to >> carbon_home]/returns: current_value notrun, should be 0 (noop) >> Notice: /Stage[main]/Agent/Exec[copy launch-params to carbon_home]: Would >> have triggered 'refresh' from 1 events >> Notice: Agent::Push_templates[extensions/clean.sh]: Would have triggered >> 'refresh' from 1 events >> Notice: Agent::Push_templates[conf/templates/jndi.properties.template]: >> Would have triggered 'refresh' from 1 events >> Notice: Agent::Push_templates[extensions/start-servers.sh]: Would have >> triggered 'refresh' from 1 events >> Notice: Agent::Push_templates[extensions/instance-started.sh]: Would have >> triggered 'refresh' from 1 events >> Notice: Agent::Push_templates[extensions/artifacts-updated.sh]: Would >> have triggered 'refresh' from 1 events >> Notice: Agent::Start[cartridge-agent]: Would have triggered 'refresh' >> from 1 events >> Info: Agent::Start[cartridge-agent]: Scheduling refresh of >> Exec[starting_cartridge-agent] >> Notice: >> /Stage[main]/Agent/Agent::Start[cartridge-agent]/Exec[starting_cartridge-agent]: >> Would have triggered 'refresh' from 1 events >> Notice: Agent::Start[cartridge-agent]: Would have triggered 'refresh' >> from 1 events >> Notice: Class[Agent]: Would have triggered 'refresh' from 4 events >> Notice: Stage[main]: Would have triggered 'refresh' from 3 events >> Notice: Finished catalog run in 3.41 seconds >> >> Please advice, Thanks >> -D >> >> On Sun, Mar 8, 2015 at 12:09 AM, david hbase <davidhb...@gmail.com> >> wrote: >> >>> After recreate the CA certificate and the puppet master certificate, the >>> above error has been fixed. >>> But the AC ( auto scaling ) still not work. Looks like the puppet agent >>> do not send back the system/health check status. >>> >>> Any advice? >>> >>> Thanks >>> -D >>> >>> >>> On Sat, Mar 7, 2015 at 8:09 PM, david hbase <davidhb...@gmail.com> >>> wrote: >>> >>>> Got another error with autoscaling, the puppet log as following: >>>> (Stratos 4.0 ) >>>> >>>> [2015-03-08 04:02:01] - -> >>>> /production/file_metadatas/modules/agent/activemq?links=manage&recurse=true&checksum_type=md5 >>>> [2015-03-08 04:02:02] 10.203.50.169 - - [08/Mar/2015:04:02:01 UTC] "PUT >>>> /production/report/phpphp201-25e.prodstage.com HTTP/1.1" 200 9 >>>> [2015-03-08 04:02:02] - -> /production/report/ >>>> phpphp201-25e.prodstage.com >>>> [2015-03-08 04:02:49] ERROR OpenSSL::SSL::SSLError: SSL_accept SYSCALL >>>> returned=5 errno=0 state=unknown state >>>> /usr/lib/ruby/vendor_ruby/puppet/network/http/webrick.rb:32:in >>>> `accept' >>>> /usr/lib/ruby/vendor_ruby/puppet/network/http/webrick.rb:32:in >>>> `listen' >>>> /usr/lib/ruby/1.8/webrick/server.rb:173:in `call' >>>> /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread' >>>> /usr/lib/ruby/1.8/webrick/server.rb:162:in `start' >>>> /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread' >>>> /usr/lib/ruby/1.8/webrick/server.rb:95:in `start' >>>> /usr/lib/ruby/1.8/webrick/server.rb:92:in `each' >>>> /usr/lib/ruby/1.8/webrick/server.rb:92:in `start' >>>> /usr/lib/ruby/1.8/webrick/server.rb:23:in `start' >>>> /usr/lib/ruby/1.8/webrick/server.rb:82:in `start' >>>> /usr/lib/ruby/vendor_ruby/puppet/network/http/webrick.rb:27:in >>>> `listen' >>>> /usr/lib/ruby/vendor_ruby/puppet/network/http/webrick.rb:26:in >>>> `initialize' >>>> /usr/lib/ruby/vendor_ruby/puppet/network/http/webrick.rb:26:in `new' >>>> /usr/lib/ruby/vendor_ruby/puppet/network/http/webrick.rb:26:in >>>> `listen' >>>> /usr/lib/ruby/vendor_ruby/puppet/network/server.rb:27:in `start' >>>> /usr/lib/ruby/vendor_ruby/puppet/daemon.rb:139:in `start' >>>> /usr/lib/ruby/vendor_ruby/puppet/application/master.rb:286:in >>>> `start_webrick_master' >>>> /usr/lib/ruby/vendor_ruby/puppet/application/master.rb:199:in `main' >>>> /usr/lib/ruby/vendor_ruby/puppet/application/master.rb:160:in >>>> `run_command' >>>> /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run' >>>> /usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook' >>>> /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run' >>>> /usr/lib/ruby/vendor_ruby/puppet/util.rb:488:in `exit_on_fail' >>>> /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run' >>>> /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run' >>>> /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in >>>> `execute' >>>> /usr/bin/puppet:8 >>>> [2015-03-08 04:02:49] ERROR OpenSSL::SSL::SSLError: SSL_accept SYSCALL >>>> returned=5 errno=0 state=unknown state >>>> >>>> Also, All the instance are auto expired after about 10 minutes and the >>>> new instances created automatically. >>>> >>>> Thanks >>>> -D >>>> >>>> On Fri, Mar 6, 2015 at 5:22 PM, Udara Liyanage <ud...@wso2.com> wrote: >>>> >>>>> Great to hear David >>>>> On 7 Mar 2015 06:42, "david hbase" <davidhb...@gmail.com> wrote: >>>>> >>>>>> Thanks you guys, The issue has been solved, following the 646 >>>>>> instruction. >>>>>> >>>>>> -D >>>>>> >>>>>> On Fri, Mar 6, 2015 at 3:12 PM, david hbase <davidhb...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Great! It's working NOW!!! >>>>>>> >>>>>>> Just additional issue, instance can spawn, but no IP address back, >>>>>>> Logs as following ( with openstack deployement) >>>>>>> >>>>>>> [2015-03-06 22:53:10,437] INFO >>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>>>>>> - Trying to spawn an instance via cloud controller: [cluster] >>>>>>> php.php.domain [partition] zone-1 [lb-cluster] null >>>>>>> [network-partition-id] >>>>>>> openstack-region1 >>>>>>> [2015-03-06 22:53:25,542] INFO >>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>>> Instance is successfully starting up. MemberContext >>>>>>> [memberId=php.php.domainee0dea8b-a7c5-4c42-ba56-45305ade31a3, >>>>>>> nodeId=us-internal-1/20971aad-2737-48ac-96f7-7f318dc0a908, >>>>>>> clusterId=php.php.domain, cartridgeType=php, privateIpAddress=null, >>>>>>> publicIpAddress=null, allocatedIpAddress=null, initTime=1425682390438, >>>>>>> lbClusterId=null, networkPartitionId=openstack-region1] >>>>>>> [2015-03-06 22:53:25,930] ERROR >>>>>>> {org.apache.stratos.cloud.controller.iaases.OpenstackNovaIaas} - >>>>>>> Failed to >>>>>>> allocate an IP address. >>>>>>> [2015-03-06 22:53:25,931] ERROR >>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>>> Error occurred while allocating an ip address. MemberContext >>>>>>> [memberId=php.php.domainee0dea8b-a7c5-4c42-ba56-45305ade31a3, >>>>>>> nodeId=us-internal-1/20971aad-2737-48ac-96f7-7f318dc0a908, >>>>>>> clusterId=php.php.domain, cartridgeType=php, privateIpAddress=null, >>>>>>> publicIpAddress=null, allocatedIpAddress=null, initTime=1425682390438, >>>>>>> lbClusterId=null, networkPartitionId=openstack-region1] >>>>>>> org.apache.stratos.cloud.controller.exception.CloudControllerException: >>>>>>> Failed to allocate an IP address. >>>>>>> at >>>>>>> org.apache.stratos.cloud.controller.iaases.OpenstackNovaIaas.associateAddress(OpenstackNovaIaas.java:259) >>>>>>> at >>>>>>> org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl$IpAllocator.run(CloudControllerServiceImpl.java:635) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>> Exception in thread "pool-31-thread-1" >>>>>>> org.apache.stratos.cloud.controller.exception.CloudControllerException: >>>>>>> Error occurred while allocating an ip address. MemberContext >>>>>>> [memberId=php.php.domainee0dea8b-a7c5-4c42-ba56-45305ade31a3, >>>>>>> nodeId=us-internal-1/20971aad-2737-48ac-96f7-7f318dc0a908, >>>>>>> clusterId=php.php.domain, cartridgeType=php, privateIpAddress=null, >>>>>>> publicIpAddress=null, allocatedIpAddress=null, initTime=1425682390438, >>>>>>> lbClusterId=null, networkPartitionId=openstack-region1] >>>>>>> at >>>>>>> org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl$IpAllocator.run(CloudControllerServiceImpl.java:694) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>> Caused by: >>>>>>> org.apache.stratos.cloud.controller.exception.CloudControllerException: >>>>>>> Failed to allocate an IP address. >>>>>>> at >>>>>>> org.apache.stratos.cloud.controller.iaases.OpenstackNovaIaas.associateAddress(OpenstackNovaIaas.java:259) >>>>>>> at >>>>>>> org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl$IpAllocator.run(CloudControllerServiceImpl.java:635) >>>>>>> ... 3 more >>>>>>> >>>>>>> Just following the https://issues.apache.org/jira/browse/STRATOS-646 >>>>>>> instruction, but still not work. >>>>>>> >>>>>>> Please help. >>>>>>> Thanks >>>>>>> -D >>>>>>> >>>>>>> >>>>>>> On Wed, Mar 4, 2015 at 10:41 PM, Udara Liyanage <ud...@wso2.com> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> On 5 Mar 2015 10:15, "david hbase" <davidhb...@gmail.com> wrote: >>>>>>>> > >>>>>>>> > Great appreciated your help. The json code has been attached: >>>>>>>> > >>>>>>>> > { >>>>>>>> > "id": "zone-1", >>>>>>>> > "provider": "openstack", >>>>>>>> > "property": [ >>>>>>>> > { >>>>>>>> > "name": "region", >>>>>>>> > "value": "us-internal-1" >>>>>>>> > } >>>>>>>> > ] >>>>>>>> > } >>>>>>>> > >>>>>>>> > >>>>>>>> > { >>>>>>>> > "type": "php", >>>>>>>> > "provider": "apache", >>>>>>>> > "host": "cartridge-php", >>>>>>>> > "displayName": "PHP", >>>>>>>> > "description": "PHP Cartridge", >>>>>>>> > "version": "8", >>>>>>>> > "multiTenant": true, >>>>>>>> > "persistance": true, >>>>>>>> > "portMapping": [ >>>>>>>> > { >>>>>>>> > "protocol": "http", >>>>>>>> > "port": "80", >>>>>>>> > "proxyPort": "8280" >>>>>>>> > } >>>>>>>> > ], >>>>>>>> > "deployment": {}, >>>>>>>> > "iaasProvider": [ >>>>>>>> > { >>>>>>>> > "type": "openstack", >>>>>>>> > "imageId": >>>>>>>> "us-internal-1/287614db-0033-4b9a-80ae-7f5e6be0a942", >>>>>>>> > "property": [ >>>>>>>> > { >>>>>>>> > "name": "instanceType", >>>>>>>> > "value": >>>>>>>> "us-internal-1/c33143d9-ce0a-472c-944b-3e39476970d3" >>>>>>>> > } >>>>>>>> > ] >>>>>>>> > } >>>>>>>> > ] >>>>>>>> > } >>>>>>>> > >>>>>>>> > { >>>>>>>> > "serviceName": "appserver", >>>>>>>> > "cartridgeType": "php", >>>>>>>> > "deploymentPolicyName": "economyDeployment", >>>>>>>> > "autoscalingPolicyName": "economyPolicy", >>>>>>>> > "clusterDomain": "stratos40build05", >>>>>>>> > "clusterSubDomain": "php", >>>>>>>> > "tenantRange": "*" >>>>>>>> > } >>>>>>>> > >>>>>>>> > [2015-03-05 01:58:39,126] INFO >>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>>>> All partitions [zone-1] were validated successfully, against the >>>>>>>> Cartridge: >>>>>>>> php >>>>>>>> > [2015-03-05 01:58:39,129] INFO >>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>>>>>>> - Validating partitions of policy via cloud controller: [id] >>>>>>>> economyDeployment >>>>>>>> > [2015-03-05 01:58:39,923] INFO >>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>>>> All partitions [zone-1] were validated successfully, against the >>>>>>>> Cartridge: >>>>>>>> php >>>>>>>> > [2015-03-05 01:59:03,234] ERROR >>>>>>>> {org.apache.stratos.manager.manager.CartridgeSubscriptionManager} - No >>>>>>>> active instances are found for cartridge [php] >>>>>>>> >>>>>>>> This error message is shown when you try to subscribe to php >>>>>>>> cartridge, but there is no php instance which has become active. Could >>>>>>>> you >>>>>>>> please check weather the php instance became active. You can check >>>>>>>> cartridge agents log. >>>>>>>> >>>>>>>> > [2015-03-05 01:59:08,566] INFO >>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>>>>>>> - Validating partitions of policy via cloud controller: [id] >>>>>>>> economyDeployment2 >>>>>>>> > [2015-03-05 01:59:11,445] INFO >>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>>>>>>> - Validating partitions of policy via cloud controller: [id] >>>>>>>> economyDeployment2 >>>>>>>> > [2015-03-05 01:59:13,100] INFO >>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>>>>>>> - Validating partitions of policy via cloud controller: [id] >>>>>>>> economyDeployment2 >>>>>>>> > [2015-03-05 01:59:14,574] INFO >>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>>>> All partitions [zone-1] were validated successfully, against the >>>>>>>> Cartridge: >>>>>>>> php >>>>>>>> > [2015-03-05 01:59:14,577] INFO >>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>>>>>>> - Validating partitions of policy via cloud controller: [id] >>>>>>>> economyDeployment >>>>>>>> > [2015-03-05 01:59:14,628] INFO >>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>>>> All partitions [zone-1] were validated successfully, against the >>>>>>>> Cartridge: >>>>>>>> php >>>>>>>> > [2015-03-05 01:59:14,631] INFO >>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>>>>>>> - Validating partitions of policy via cloud controller: [id] >>>>>>>> economyDeployment >>>>>>>> > [2015-03-05 01:59:15,582] INFO >>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>>>> All partitions [zone-1] were validated successfully, against the >>>>>>>> Cartridge: >>>>>>>> php >>>>>>>> > [2015-03-05 01:59:15,585] INFO >>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>>>>>>> - Validating partitions of policy via cloud controller: [id] >>>>>>>> economyDeployment >>>>>>>> > [2015-03-05 01:59:15,664] INFO >>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>>>> All partitions [zone-1] were validated successfully, against the >>>>>>>> Cartridge: >>>>>>>> php >>>>>>>> > [2015-03-05 01:59:16,549] INFO >>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>>>> All partitions [zone-1] were validated successfully, against the >>>>>>>> Cartridge: >>>>>>>> php >>>>>>>> > [2015-03-05 01:59:16,626] INFO >>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>>>> All partitions [zone-1] were validated successfully, against the >>>>>>>> Cartridge: >>>>>>>> php >>>>>>>> > >>>>>>>> > Thanks >>>>>>>> > -D >>>>>>>> > >>>>>>>> > On Wed, Mar 4, 2015 at 6:13 PM, Udara Liyanage <ud...@wso2.com> >>>>>>>> wrote: >>>>>>>> >> >>>>>>>> >> Hi, >>>>>>>> >> >>>>>>>> >> First of all php is normally not considered a multi tenant >>>>>>>> service. However could you share your php cartridge json and service >>>>>>>> json >>>>>>>> which threw the exception. >>>>>>>> >> >>>>>>>> >> On 5 Mar 2015 05:21, "david hbase" <davidhb...@gmail.com> wrote: >>>>>>>> >>> >>>>>>>> >>> Hi team >>>>>>>> >>> >>>>>>>> >>> With Stratos 4.0 and Openstack, I have deployed all the steps >>>>>>>> successfully, only except with the last step for the Multi-Tenant >>>>>>>> Service >>>>>>>> failed, and the error log as following: >>>>>>>> >>> >>>>>>>> >>> Failed to deploy the Service [Cartridge type] php [alias] >>>>>>>> a92a1da2-57bf-4e86-89a3-146900993793 . Cause: null >>>>>>>> >>> >>>>>>>> >>> Please Advice, waiting for your reply. >>>>>>>> >>> >>>>>>>> >>> Thanks >>>>>>>> >>> -D >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> [2015-03-04 23:22:57,413] INFO >>>>>>>> {org.apache.stratos.rest.endpoint.services.ServiceUtils} - Deploying >>>>>>>> service.. >>>>>>>> >>> [2015-03-04 23:22:57,434] INFO >>>>>>>> {org.apache.stratos.manager.subscription.utils.CartridgeSubscriptionUtils} >>>>>>>> - Generated key : AYhQR4NuUnTFVtCI >>>>>>>> >>> [2015-03-04 23:22:57,436] ERROR >>>>>>>> {org.apache.stratos.rest.endpoint.services.ServiceUtils} - Failed to >>>>>>>> deploy the Service [Cartridge type] php [alias] >>>>>>>> 0405644c-c371-4201-a05c-b24c337d7896 . Cause: null >>>>>>>> >>> java.lang.NullPointerException >>>>>>>> >>> at >>>>>>>> org.apache.stratos.manager.deploy.service.ServiceDeploymentManager.deployService(ServiceDeploymentManager.java:322) >>>>>>>> >>> at >>>>>>>> org.apache.stratos.rest.endpoint.services.ServiceUtils.deployService(ServiceUtils.java:1133) >>>>>>>> >>> at >>>>>>>> org.apache.stratos.rest.endpoint.services.StratosAdmin.deployService(StratosAdmin.java:908) >>>>>>>> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>> >>> at >>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>>>> >>> at >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>> >>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>> >>> at >>>>>>>> org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180) >>>>>>>> >>> at >>>>>>>> org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) >>>>>>>> >>> at >>>>>>>> org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:194) >>>>>>>> >>> at >>>>>>>> org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:100) >>>>>>>> >>> at >>>>>>>> org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57) >>>>>>>> >>> at >>>>>>>> org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93) >>>>>>>> >>> at >>>>>>>> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271) >>>>>>>> >>> at >>>>>>>> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) >>>>>>>> >>> at >>>>>>>> org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239) >>>>>>>> >>> at >>>>>>>> org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:223) >>>>>>>> >>> at >>>>>>>> org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:203) >>>>>>>> >>> at >>>>>>>> org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:137) >>>>>>>> >>> at >>>>>>>> org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:159) >>>>>>>> >>> at >>>>>>>> org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286) >>>>>>>> >>> at >>>>>>>> org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:206) >>>>>>>> >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755) >>>>>>>> >>> at >>>>>>>> org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262) >>>>>>>> >>> at >>>>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) >>>>>>>> >>> at >>>>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >>>>>>>> >>> at >>>>>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) >>>>>>>> >>> at >>>>>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) >>>>>>>> >>> at >>>>>>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) >>>>>>>> >>> at >>>>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) >>>>>>>> >>> at >>>>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) >>>>>>>> >>> at >>>>>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178) >>>>>>>> >>> at >>>>>>>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47) >>>>>>>> >>> at >>>>>>>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56) >>>>>>>> >>> at >>>>>>>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47) >>>>>>>> >>> at >>>>>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141) >>>>>>>> >>> at >>>>>>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156) >>>>>>>> >>> at >>>>>>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936) >>>>>>>> >>> at >>>>>>>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52) >>>>>>>> >>> at >>>>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) >>>>>>>> >>> at >>>>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) >>>>>>>> >>> at >>>>>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004) >>>>>>>> >>> at >>>>>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) >>>>>>>> >>> at >>>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653) >>>>>>>> >>> at >>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>> >>> at >>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>> >>> at java.lang.Thread.run(Thread.java:745) >>>>>>>> >>> >>>>>>>> > >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >>> >> > > > -- > Rajkumar Rajaratnam > Committer & PMC Member, Apache Stratos > Software Engineer, WSO2 > > Mobile : +94777568639 > Blog : rajkumarr.com >