Hi Manula, We found the issue and fixed it. Problem is updating TimeStampMap without checking whether the member is in the topology or not.
We are removing member from the map on member terminated event. Also we are updating the map on receiving health stats message from members. There can be situation where the health stat message comes bit later than the member terminated event. In this situation first we are removing member from map but again we are updating the time stamp map on receiving health stats messages. It means CEP is still considering it as a active member. We solved it by checking whether the member is in the topology before updating time stamp map. Thanks. On Mon, Oct 13, 2014 at 2:51 PM, Manula Chathurika Thantriwatte < manu...@wso2.com> wrote: > Hi Raj, > > It seems like [1] get executed again. memberTimeStampMap is a concurrent > hash map. Seems that CEPTopologyEventReceiver unable to remove the element > in the memberTimeStampMap. > > [1] > https://github.com/apache/stratos/blob/master/extensions/cep/stratos-cep-extension/src/main/java/org/apache/stratos/cep/extension/FaultHandlingWindowProcessor.java#L220 > > Thanks ! > > On Mon, Oct 13, 2014 at 11:48 AM, Rajkumar Rajaratnam <rajkum...@wso2.com> > wrote: > >> Hi, >> >> CEP extension FaultHandlingWindowProcessor removing a terminated member >> from the map. But after a while, it is complaining it as a faulty member. >> Here is the log. MemberId highlighted. >> >> TID: [0] [STRATOS] [2014-10-13 11:15:22,409] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the >> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time >> intervals]: 1 >> TID: [0] [STRATOS] [2014-10-13 11:15:22,410] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the >> value, [average]: 41.63388 , [gradient]: -0.013520682 , [second >> derivative]: 0.0013238431 , [time intervals]: 1 >> TID: [0] [STRATOS] [2014-10-13 11:15:22,410] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the >> value, [average]: 99.9 , [gradient]: -2.0885546 , [second derivative]: >> 0.03460533 , [time intervals]: 1 >> TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [Replicas] minReplicas : 2 >> TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [Replicas] maxReplicas : 20 >> TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [Replicas] nonTerminated : 5 >> TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [Replicas] activeReplicas : 5 >> TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [RequestInFlight] resetted ? : false >> TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [RequestInFlight] predicted value : 0.0 >> TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [RequestInFlight] upper limit : 80.0 >> TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [RequestInFlight] lower limit : 5.0 >> TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [MemoryConsumption] resetted ? : true >> TID: [0] [STRATOS] [2014-10-13 11:15:22,413] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [MemoryConsumption] predicted value : 41.62102324969601 >> TID: [0] [STRATOS] [2014-10-13 11:15:22,413] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [MemoryConsumption] upper limit : 80.0 >> TID: [0] [STRATOS] [2014-10-13 11:15:22,413] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [MemoryConsumption] lower limit : 15.0 >> TID: [0] [STRATOS] [2014-10-13 11:15:22,415] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [LoadAverage] resetted ? : true >> TID: [0] [STRATOS] [2014-10-13 11:15:22,415] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [LoadAverage] predicted value : 97.82874980941415 >> TID: [0] [STRATOS] [2014-10-13 11:15:22,415] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [LoadAverage] upper limit : 180.0 >> TID: [0] [STRATOS] [2014-10-13 11:15:22,415] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [LoadAverage] lower limit : 20.0 >> TID: [0] [STRATOS] [2014-10-13 11:15:22,416] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain scale-up action : false >> TID: [0] [STRATOS] [2014-10-13 11:15:22,416] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain scale-down action : true >> TID: [0] [STRATOS] [2014-10-13 11:15:22,416] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to >> scale-down [cluster] : myphp.php.domain >> TID: [0] [STRATOS] [2014-10-13 11:15:22,417] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-down] [cluster] : >> myphp.php.domain shrink the cluster to minReplicas : 2 >> TID: [0] [STRATOS] [2014-10-13 11:15:22,417] INFO >> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >> - Updating kubernetes replication controller via cloud controller: >> [cluster] myphp.php.domain [replicas] 2 >> TID: [0] [STRATOS] [2014-10-13 11:15:22,425] DEBUG >> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >> CloudControllerServiceImpl:updateContainers for cluster : myphp.php.domain >> TID: [0] [STRATOS] [2014-10-13 11:15:22,448] DEBUG >> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >> Cloud Controller is delegating request to update a replication controller >> myphp.php.domain to Kubernetes layer. >> TID: [0] [STRATOS] [2014-10-13 11:15:22,506] DEBUG >> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >> Cloud Controller successfully updated the controller myphp.php.domain via >> Kubernetes layer. >> TID: [0] [STRATOS] [2014-10-13 11:15:22,531] DEBUG >> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >> Pods Count: 5 for cluster: myphp.php.domain >> TID: [0] [STRATOS] [2014-10-13 11:15:22,548] DEBUG >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> PodActivationWatcher running : Running >> TID: [0] [STRATOS] [2014-10-13 11:15:22,561] DEBUG >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> PodActivationWatcher running : Running >> TID: [0] [STRATOS] [2014-10-13 11:15:22,563] DEBUG >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> PodActivationWatcher running : Running >> TID: [0] [STRATOS] [2014-10-13 11:15:22,655] DEBUG >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> PodActivationWatcher running : Running >> TID: [0] [STRATOS] [2014-10-13 11:15:22,657] DEBUG >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> PodActivationWatcher running : Running >> TID: [0] [STRATOS] [2014-10-13 11:15:27,538] DEBUG >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> PodActivationWatcher running : Running >> TID: [0] [STRATOS] [2014-10-13 11:15:27,539] ERROR >> {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod >> [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist. >> TID: [0] [STRATOS] [2014-10-13 11:15:27,539] ERROR >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> Container Activation Watcher Failed.. Cause: Pod >> [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist. >> TID: [0] [STRATOS] [2014-10-13 11:15:27,539] DEBUG >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException: >> Pod [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist. >> TID: [0] [STRATOS] [2014-10-13 11:15:27,539] ERROR >> {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod >> [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist. >> TID: [0] [STRATOS] [2014-10-13 11:15:27,540] ERROR >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> Container Activation Watcher Failed.. Cause: Pod >> [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist. >> TID: [0] [STRATOS] [2014-10-13 11:15:27,540] DEBUG >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException: >> Pod [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist. >> TID: [0] [STRATOS] [2014-10-13 11:15:27,628] ERROR >> {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod >> [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist. >> TID: [0] [STRATOS] [2014-10-13 11:15:27,628] ERROR >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> Container Activation Watcher Failed.. Cause: Pod >> [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist. >> TID: [0] [STRATOS] [2014-10-13 11:15:27,628] DEBUG >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException: >> Pod [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist. >> TID: [0] [STRATOS] [2014-10-13 11:15:27,630] DEBUG >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> PodActivationWatcher running : Running >> TID: [0] [STRATOS] [2014-10-13 11:15:32,542] ERROR >> {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod >> [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist. >> TID: [0] [STRATOS] [2014-10-13 11:15:32,542] ERROR >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> Container Activation Watcher Failed.. Cause: Pod >> [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist. >> TID: [0] [STRATOS] [2014-10-13 11:15:32,542] DEBUG >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException: >> Pod [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist. >> TID: [0] [STRATOS] [2014-10-13 11:15:32,549] DEBUG >> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >> Pods Count: 2 for cluster: myphp.php.domain >> TID: [0] [STRATOS] [2014-10-13 11:15:32,549] DEBUG >> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >> Pods created : 2 for cluster : myphp.php.domain >> TID: [0] [STRATOS] [2014-10-13 11:15:32,551] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write >> lock acquired >> TID: [0] [STRATOS] [2014-10-13 11:15:32,551] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Updating >> topology >> TID: [0] [STRATOS] [2014-10-13 11:15:32,553] DEBUG >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> PodActivationWatcher running : Running >> TID: [0] [STRATOS] [2014-10-13 11:15:32,554] ERROR >> {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod >> [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist. >> TID: [0] [STRATOS] [2014-10-13 11:15:32,554] ERROR >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> Container Activation Watcher Failed.. Cause: Pod >> [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist. >> TID: [0] [STRATOS] [2014-10-13 11:15:32,555] DEBUG >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException: >> Pod [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist. >> TID: [0] [STRATOS] [2014-10-13 11:15:32,564] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology >> updated: >> {"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","clusterIdClusterMap":{"myphp.php.domain":{"serviceName":"php","clusterId":"myphp.php.domain","autoscalePolicyName":"economy","hostNames":[" >> myphp.apachestratos.org >> "],"isLbCluster":false,"isKubernetesCluster":true,"memberMap":{"3ccf4e4e-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccf4e4e-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}},"b446c0dd-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b446c0dd-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"b44650cb-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b44650cb-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"3ccfb8bd-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccfb8bd-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}}},"status":"Created","properties":{"payload_parameter.APP_PATH":"/var/www","KUBERNETES_REPLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"KubGrp1","payload_parameter.LOG_LEVEL":"DEBUG","KUBERNETES_REPLICAS_MIN":"2","payload_parameter.MB_IP":"172.17.42.1","payload_parameter.CEP_IP":"172.17.42.1","payload_parameter.MB_PORT":"1883","payload_parameter.CEP_PORT":"7611"}}},"portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"properties":{}}},"initialized":false} >> TID: [0] [STRATOS] [2014-10-13 11:15:32,564] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write >> lock released >> TID: [0] [STRATOS] [2014-10-13 11:15:32,564] INFO >> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - >> Publishing >> member terminated event: [service] php [cluster] myphp.php.domain >> [network-partition] null [partition] null [member] >> b4468fb6-529b-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:15:32,575] INFO >> {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver} >> - [MemberTerminatedEventListener] Received: class >> org.apache.stratos.messaging.event.topology.MemberTerminatedEvent >> TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG >> {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver} >> - Removed the terminated member with id >> b4468fb6-529b-11e4-acbb-08002794b041 from the cluster >> TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG >> {org.apache.stratos.manager.topology.model.TopologyClusterInformationModel} >> - Adding cluster [myphp.php.domain] >> TID: [0] [STRATOS] [2014-10-13 11:15:32,580] INFO >> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} >> - Member terminated: [service] php [cluster] myphp.php.domain [member] >> b4468fb6-529b-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG >> {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - >> MemberContext [id]: b4468fb6-529b-11e4-acbb-08002794b041 removed from >> information model. >> TID: [0] [STRATOS] [2014-10-13 11:15:32,581] INFO >> {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - >> Scheduled Pod Activation Watcher task canceled. >> TID: [0] [STRATOS] [2014-10-13 11:15:32,581] WARN >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member >> is in the wrong list and it is removed from active members list >> TID: [0] [STRATOS] [2014-10-13 11:15:32,582] INFO >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member >> stat context has been removed successfully: [member] >> b4468fb6-529b-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:15:32,582] WARN >> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} >> - Member already terminated: [service] php [cluster] myphp.php.domain >> [member] b4468fb6-529b-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:15:32,583] INFO >> {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Member >> [member id] b4468fb6-529b-11e4-acbb-08002794b041 was removed from the time >> stamp map. >> TID: [0] [STRATOS] [2014-10-13 11:15:32,583] WARN >> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} >> - Member already terminated: [service] php [cluster] myphp.php.domain >> [member] b4468fb6-529b-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:15:32,593] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write >> lock acquired >> TID: [0] [STRATOS] [2014-10-13 11:15:32,593] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Updating >> topology >> TID: [0] [STRATOS] [2014-10-13 11:15:32,603] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology >> updated: >> {"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","clusterIdClusterMap":{"myphp.php.domain":{"serviceName":"php","clusterId":"myphp.php.domain","autoscalePolicyName":"economy","hostNames":[" >> myphp.apachestratos.org >> "],"isLbCluster":false,"isKubernetesCluster":true,"memberMap":{"3ccf4e4e-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccf4e4e-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}},"b44650cb-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b44650cb-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"3ccfb8bd-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccfb8bd-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}}},"status":"Created","properties":{"payload_parameter.APP_PATH":"/var/www","KUBERNETES_REPLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"KubGrp1","payload_parameter.LOG_LEVEL":"DEBUG","KUBERNETES_REPLICAS_MIN":"2","payload_parameter.MB_IP":"172.17.42.1","payload_parameter.CEP_IP":"172.17.42.1","payload_parameter.MB_PORT":"1883","payload_parameter.CEP_PORT":"7611"}}},"portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"properties":{}}},"initialized":false} >> TID: [0] [STRATOS] [2014-10-13 11:15:32,603] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write >> lock released >> TID: [0] [STRATOS] [2014-10-13 11:15:32,603] INFO >> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - >> Publishing member terminated event: [service] php [cluster] >> myphp.php.domain [network-partition] null [partition] null [member] >> b446c0dd-529b-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:15:32,607] INFO >> {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Member >> [member id] b446c0dd-529b-11e4-acbb-08002794b041 was removed from the time >> stamp map. >> TID: [0] [STRATOS] [2014-10-13 11:15:32,608] INFO >> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} >> - Member terminated: [service] php [cluster] myphp.php.domain [member] >> b446c0dd-529b-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:15:32,609] INFO >> {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver} >> - [MemberTerminatedEventListener] Received: class >> org.apache.stratos.messaging.event.topology.MemberTerminatedEvent >> TID: [0] [STRATOS] [2014-10-13 11:15:32,609] DEBUG >> {org.apache.stratos.manager.topology.model.TopologyClusterInformationModel} >> - Adding cluster [myphp.php.domain] >> TID: [0] [STRATOS] [2014-10-13 11:15:32,609] WARN >> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} >> - Member already terminated: [service] php [cluster] myphp.php.domain >> [member] b446c0dd-529b-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:15:32,609] WARN >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member >> is in the wrong list and it is removed from active members list >> TID: [0] [STRATOS] [2014-10-13 11:15:32,609] INFO >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member >> stat context has been removed successfully: [member] >> b446c0dd-529b-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:15:32,610] DEBUG >> {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - >> MemberContext [id]: b446c0dd-529b-11e4-acbb-08002794b041 removed from >> information model. >> TID: [0] [STRATOS] [2014-10-13 11:15:32,610] WARN >> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} >> - Member already terminated: [service] php [cluster] myphp.php.domain >> [member] b446c0dd-529b-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:15:32,610] INFO >> {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - >> Scheduled Pod Activation Watcher task canceled. >> TID: [0] [STRATOS] [2014-10-13 11:15:32,623] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write >> lock acquired >> TID: [0] [STRATOS] [2014-10-13 11:15:32,624] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Updating >> topology >> TID: [0] [STRATOS] [2014-10-13 11:15:32,627] ERROR >> {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod >> [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist. >> TID: [0] [STRATOS] [2014-10-13 11:15:32,627] ERROR >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> Container Activation Watcher Failed.. Cause: Pod >> [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist. >> TID: [0] [STRATOS] [2014-10-13 11:15:32,628] DEBUG >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException: >> Pod [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist. >> TID: [0] [STRATOS] [2014-10-13 11:15:32,633] DEBUG >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> PodActivationWatcher running : Running >> TID: [0] [STRATOS] [2014-10-13 11:15:32,642] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology >> updated: >> {"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","clusterIdClusterMap":{"myphp.php.domain":{"serviceName":"php","clusterId":"myphp.php.domain","autoscalePolicyName":"economy","hostNames":[" >> myphp.apachestratos.org >> "],"isLbCluster":false,"isKubernetesCluster":true,"memberMap":{"b44650cb-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b44650cb-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"3ccfb8bd-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccfb8bd-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}}},"status":"Created","properties":{"payload_parameter.APP_PATH":"/var/www","KUBERNETES_REPLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"KubGrp1","payload_parameter.LOG_LEVEL":"DEBUG","KUBERNETES_REPLICAS_MIN":"2","payload_parameter.MB_IP":"172.17.42.1","payload_parameter.CEP_IP":"172.17.42.1","payload_parameter.MB_PORT":"1883","payload_parameter.CEP_PORT":"7611"}}},"portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"properties":{}}},"initialized":false} >> TID: [0] [STRATOS] [2014-10-13 11:15:32,642] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write >> lock released >> TID: [0] [STRATOS] [2014-10-13 11:15:32,642] INFO >> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - >> Publishing member terminated event: [service] php [cluster] >> myphp.php.domain [network-partition] null [partition] null [member] >> 3ccf4e4e-529a-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:15:32,650] INFO >> {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Member >> [member id] 3ccf4e4e-529a-11e4-acbb-08002794b041 was removed from the time >> stamp map. >> TID: [0] [STRATOS] [2014-10-13 11:15:32,652] INFO >> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} >> - Member terminated: [service] php [cluster] myphp.php.domain [member] >> 3ccf4e4e-529a-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:15:32,653] INFO >> {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver} >> - [MemberTerminatedEventListener] Received: class >> org.apache.stratos.messaging.event.topology.MemberTerminatedEvent >> TID: [0] [STRATOS] [2014-10-13 11:15:32,653] DEBUG >> {org.apache.stratos.manager.topology.model.TopologyClusterInformationModel} >> - Adding cluster [myphp.php.domain] >> TID: [0] [STRATOS] [2014-10-13 11:15:32,653] WARN >> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} >> - Member already terminated: [service] php [cluster] myphp.php.domain >> [member] 3ccf4e4e-529a-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:15:32,654] WARN >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member >> is in the wrong list and it is removed from active members list >> TID: [0] [STRATOS] [2014-10-13 11:15:32,654] DEBUG >> {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - >> MemberContext [id]: 3ccf4e4e-529a-11e4-acbb-08002794b041 removed from >> information model. >> TID: [0] [STRATOS] [2014-10-13 11:15:32,654] INFO >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member >> stat context has been removed successfully: [member] >> 3ccf4e4e-529a-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:15:32,655] WARN >> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} >> - Member already terminated: [service] php [cluster] myphp.php.domain >> [member] 3ccf4e4e-529a-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:15:32,654] INFO >> {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - >> Scheduled Pod Activation Watcher task canceled. >> TID: [0] [STRATOS] [2014-10-13 11:15:32,696] INFO >> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >> Kubernetes entities are successfully starting up. [MemberContext >> [memberId=b4468fb6-529b-11e4-acbb-08002794b041, nodeId=null, >> instanceId=null, clusterId=myphp.php.domain, partition=null, >> cartridgeType=php, privateIpAddress=172.17.8.100, >> publicIpAddress=172.17.8.100, allocatedIpAddress=null, >> initTime=1413178952532, lbClusterId=null, networkPartitionId=null, >> properties=Properties [properties=[Property >> [name=ALLOCATED_SERVICE_HOST_PORT, value=null]]]], MemberContext >> [memberId=b446c0dd-529b-11e4-acbb-08002794b041, nodeId=null, >> instanceId=null, clusterId=myphp.php.domain, partition=null, >> cartridgeType=php, privateIpAddress=172.17.8.101, >> publicIpAddress=172.17.8.101, allocatedIpAddress=null, >> initTime=1413178952532, lbClusterId=null, networkPartitionId=null, >> properties=Properties [properties=[Property >> [name=ALLOCATED_SERVICE_HOST_PORT, value=null]]]], MemberContext >> [memberId=3ccf4e4e-529a-11e4-acbb-08002794b041, nodeId=null, >> instanceId=null, clusterId=myphp.php.domain, partition=null, >> cartridgeType=php, privateIpAddress=172.17.8.101, >> publicIpAddress=172.17.8.101, allocatedIpAddress=null, >> initTime=1413178322623, lbClusterId=null, networkPartitionId=null, >> properties=Properties [properties=[Property >> [name=ALLOCATED_SERVICE_HOST_PORT, value=4500]]]]] >> TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling down, >> terminated the member with id b4468fb6-529b-11e4-acbb-08002794b041 in >> cluster myphp.php.domain >> TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling down, >> terminated the member with id b446c0dd-529b-11e4-acbb-08002794b041 in >> cluster myphp.php.domain >> TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling down, >> terminated the member with id 3ccf4e4e-529a-11e4-acbb-08002794b041 in >> cluster myphp.php.domain >> TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG >> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Scale check >> executed for : >> org.apache.stratos.autoscaler.KubernetesClusterContext@29d32c84 >> TID: [0] [STRATOS] [2014-10-13 11:16:01,002] DEBUG >> {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishing >> complete tenant event >> TID: [0] [STRATOS] [2014-10-13 11:16:01,006] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} - >> Executing topology synchronization task >> TID: [0] [STRATOS] [2014-10-13 11:16:01,006] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - >> Publishing complete topology event >> TID: [0] [STRATOS] [2014-10-13 11:16:22,230] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >> load avg event: [cluster] myphp.php.domain [value] -0.70397747 >> TID: [0] [STRATOS] [2014-10-13 11:16:22,232] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >> Memory Consumption event: [cluster] myphp.php.domain [value] -0.46528494 >> TID: [0] [STRATOS] [2014-10-13 11:16:22,234] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >> Memory Consumption event: [cluster] myphp.php.domain [value] 27.538944 >> TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >> Derivation of load avg event: [cluster] myphp.php.domain [value] >> -0.0012131054 >> TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >> Derivation of Memory Consumption event: [cluster] myphp.php.domain [value] >> 0.021650476 >> TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-13 11:16:22,237] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >> load avg event: [cluster] myphp.php.domain [value] 53.7 >> TID: [0] [STRATOS] [2014-10-13 11:16:22,237] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-13 11:16:22,425] DEBUG >> {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver} >> - Member not found in the Topology: [member] >> 3ccf4e4e-529a-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:16:22,426] DEBUG >> {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver} >> - Member not found in the Topology: [member] >> 3ccf4e4e-529a-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:16:22,426] DEBUG >> {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver} >> - Member not found in the Topology: [member] >> 3ccf4e4e-529a-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:16:22,427] DEBUG >> {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver} >> - Member not found in the Topology: [member] >> 3ccf4e4e-529a-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:16:22,427] DEBUG >> {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver} >> - Member not found in the Topology: [member] >> 3ccf4e4e-529a-11e4-acbb-08002794b041 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,406] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - >> KubernetesServiceClusterMonitor is running.. >> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1, >> clusterId=myphp.php.domain, serviceId=php] >> TID: [0] [STRATOS] [2014-10-13 11:16:52,407] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - >> Running min check for [kub-cluster] : KubGrp1 [cluster] : myphp.php.domain >> TID: [0] [STRATOS] [2014-10-13 11:16:52,408] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule: >> [kub-cluster] KubGrp1 [cluster] myphp.php.domain >> TID: [0] [STRATOS] [2014-10-13 11:16:52,408] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] : >> myphp.php.domain [Replicas] nonTerminated : 2 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,409] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] : >> myphp.php.domain [Replicas] minReplicas : 2 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,409] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers >> rule [kub-cluster] : KubGrp1 [cluster] : myphp.php.domain >> TID: [0] [STRATOS] [2014-10-13 11:16:52,410] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster] >> : myphp.php.domain [Replicas] obsoleteReplicas : 0 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,411] DEBUG >> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum >> check executed for : >> org.apache.stratos.autoscaler.KubernetesClusterContext@29d32c84 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,411] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - >> flag of rifReset : false flag of memoryConsumptionReset : true flag of >> loadAverageReset : true >> TID: [0] [STRATOS] [2014-10-13 11:16:52,412] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - >> Running scale check for [kub-cluster] : KubGrp1 [cluster] : >> myphp.php.domain >> TID: [0] [STRATOS] [2014-10-13 11:16:52,412] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule >> [kub-cluster] : KubGrp1 [cluster] : myphp.php.domain >> TID: [0] [STRATOS] [2014-10-13 11:16:52,414] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the >> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time >> intervals]: 1 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,415] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the >> value, [average]: 27.538944 , [gradient]: -0.46528494 , [second >> derivative]: 0.021650476 , [time intervals]: 1 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,416] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the >> value, [average]: 53.7 , [gradient]: -0.70397747 , [second derivative]: >> -0.0012131054 , [time intervals]: 1 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [Replicas] minReplicas : 2 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [Replicas] maxReplicas : 20 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [Replicas] nonTerminated : 2 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [Replicas] activeReplicas : 2 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,417] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [RequestInFlight] resetted ? : false >> TID: [0] [STRATOS] [2014-10-13 11:16:52,417] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [RequestInFlight] predicted value : 0.0 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [RequestInFlight] upper limit : 80.0 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [RequestInFlight] lower limit : 5.0 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [MemoryConsumption] resetted ? : true >> TID: [0] [STRATOS] [2014-10-13 11:16:52,418] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [MemoryConsumption] predicted value : 27.084485135041177 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,418] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [MemoryConsumption] upper limit : 80.0 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,419] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [MemoryConsumption] lower limit : 15.0 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,419] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [LoadAverage] resetted ? : true >> TID: [0] [STRATOS] [2014-10-13 11:16:52,419] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [LoadAverage] predicted value : 52.9954185327515 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,419] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [LoadAverage] upper limit : 180.0 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,420] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain [LoadAverage] lower limit : 20.0 >> TID: [0] [STRATOS] [2014-10-13 11:16:52,420] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain scale-up action : false >> TID: [0] [STRATOS] [2014-10-13 11:16:52,420] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain scale-down action : true >> TID: [0] [STRATOS] [2014-10-13 11:16:52,421] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> myphp.php.domain Even if all predicted stats < upperLimit, still >> activeReplicas <= minReplicas ; hence not scaling down >> TID: [0] [STRATOS] [2014-10-13 11:16:52,421] DEBUG >> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Scale check >> executed for : >> org.apache.stratos.autoscaler.KubernetesClusterContext@29d32c84 >> TID: [0] [STRATOS] [2014-10-13 11:17:01,001] DEBUG >> {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishing >> complete tenant event >> TID: [0] [STRATOS] [2014-10-13 11:17:01,002] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} - >> Executing topology synchronization task >> TID: [0] [STRATOS] [2014-10-13 11:17:01,002] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - >> Publishing complete topology event >> TID: [0] [STRATOS] [2014-10-13 11:17:16,141] INFO >> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty >> member detected [member-id] 3ccf4e4e-529a-11e4-acbb-08002794b041 with [last >> time-stamp] 1413179137678 [time-out] 60000 milliseconds >> TID: [0] [STRATOS] [2014-10-13 11:17:16,141] ERROR >> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Failed >> to publish member fault event. Member having [member-id] >> 3ccf4e4e-529a-11e4-acbb-08002794b041 does not exist in topology >> TID: [0] [STRATOS] [2014-10-13 11:17:22,231] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >> load avg event: [cluster] myphp.php.domain [value] 23.75 >> TID: [0] [STRATOS] [2014-10-13 11:17:22,232] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >> load avg event: [cluster] myphp.php.domain [value] -0.2682943 >> TID: [0] [STRATOS] [2014-10-13 11:17:22,236] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >> Memory Consumption event: [cluster] myphp.php.domain [value] 17.602432 >> TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >> Derivation of load avg event: [cluster] myphp.php.domain [value] >> -0.0045045386 >> TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >> Memory Consumption event: [cluster] myphp.php.domain [value] 0.0016795773 >> TID: [0] [STRATOS] [2014-10-13 11:17:22,238] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >> Derivation of Memory Consumption event: [cluster] myphp.php.domain [value] >> 2.3589536E-4 >> TID: [0] [STRATOS] [2014-10-13 11:17:22,238] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-13 11:18:01,000] DEBUG >> {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishing >> complete tenant event >> TID: [0] [STRATOS] [2014-10-13 11:18:01,001] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} - >> Executing topology synchronization task >> TID: [0] [STRATOS] [2014-10-13 11:18:01,001] DEBUG >> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - >> Publishing complete topology event >> TID: [0] [STRATOS] [2014-10-13 11:18:16,142] INFO >> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty >> member detected [member-id] 3ccf4e4e-529a-11e4-acbb-08002794b041 with [last >> time-stamp] 1413179137678 [time-out] 60000 milliseconds >> TID: [0] [STRATOS] [2014-10-13 11:18:16,144] ERROR >> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Failed >> to publish member fault event. Member having [member-id] >> 3ccf4e4e-529a-11e4-acbb-08002794b041 does not exist in topology >> >> Thanks. >> >> -- >> Rajkumar Rajaratnam >> Software Engineer | WSO2, Inc. >> Mobile +94777568639 | +94783498120 >> > > > > -- > Regards, > Manula Chathurika Thantriwatte > Software Engineer > WSO2 Inc. : http://wso2.com > lean . enterprise . middleware > > email : manu...@wso2.com / man...@apache.org > phone : +94 772492511 > blog : http://manulachathurika.blogspot.com/ > > > > -- Rajkumar Rajaratnam Software Engineer | WSO2, Inc. Mobile +94777568639 | +94783498120
