Please find the Jira https://issues.apache.org/jira/browse/STRATOS-939
I've committed the fix.
On Wed, Nov 5, 2014 at 9:02 PM, Nirmal Fernando <nirmal070...@gmail.com>
wrote:
> Hi Guys,
>
> So, I got a chance to reproduce and analyze this issue.
>
> How we calculate gradient of two events?
>
> Say the events are; e1(t1,v1) and e2(t2,v2)
>
> tx - time in *milliseconds* when xth event occurred
> vx - value (memory, cpu etc.) that xth event carries
>
> *time gap = t(2-1) = t2 -t1 milliseconds*
> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>
> Hence,
> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 - t1)*
>
> I've enabled debug logs for CEP extension;
>
> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>
> Please find the following 3 logs extracted from the debug logs;
>
> ===================================================================
> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time Gap:
> 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>
> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
> Gradient:
> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time Gap:
> 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>
> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2: 1415213232154
> hash: 155426542
> ===================================================================
>
> So, as you can see the reason behind a large value is when the time gap
> between two subjected events is less than 1 second. This could happen since
> events are coming from different asynchronous agents and also when there
> are less number of events.
>
> So, the fix I propose is a very simple one and it will not compromise
> anything AFAIS.
>
> Fix is to calculate time gap as follows;
> *____ t2 - t1*
> *| yes?*
> *time gap = t(2-1) = t2 -t1 > 1000 ---*
> *|____ 1000*
>
> I have tested this and works fine.
>
>
>
> ---------- Forwarded message ----------
> From: Manula Chathurika Thantriwatte <manu...@wso2.com>
> Date: Tue, Oct 28, 2014 at 5:16 AM
> Subject: Re: CEP sends very large values for gradient and second
> derivative of load average
> To: "dev@stratos.apache.org" <dev@stratos.apache.org>
>
>
> Hi Raj,
>
> Yes it has. But it's very easy to find the values from the agent with the
> time stamp. Then we can search that time stamp in the CEP trace logs and
> compare the summarized values.
>
> Thanks !
>
> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <rajkum...@wso2.com>
> wrote:
>
>>
>>
>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <
>> manu...@wso2.com> wrote:
>>
>>> Hi Raj,
>>>
>>> Shall we enable the cartridge agent debug logs. Then we can compare both
>>> CEP and agent logs with the time stamp. Then we can narrow down where the
>>> actual problem is.
>>>
>>
>> Manula, cep-trace log have the values sent by cartridge agent also right?
>>
>>
>>>
>>> Thanks !
>>>
>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <rajkum...@wso2.com
>>> > wrote:
>>>
>>>>
>>>>
>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
>>>> manu...@wso2.com> wrote:
>>>>
>>>>> Hi Raj,
>>>>>
>>>>> Is this happen frequently or intermittently ? Once I go though the
>>>>> logs I'm unable to find the high input values for load average. But output
>>>>> have high values.
>>>>>
>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>> JMSOutputAdaptor, sent
>>>>>
>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>
>>>>
>>>> This is happening intermittently. Experienced now too. This is a
>>>> serious issue. We need to somehow sort this out.
>>>>
>>>>
>>>>>
>>>>> Also networkPartitionId is empty. In the current execution plans it
>>>>> took networkPartitionId for the calculation. IMO we need to update the CEP
>>>>> execution plans for 4.1.0.
>>>>>
>>>>> Thanks !
>>>>>
>>>>>
>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte <
>>>>> manu...@wso2.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Thanks Raj. I'll go through them.
>>>>>>
>>>>>> Thanks !
>>>>>>
>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>> rajkum...@wso2.com> wrote:
>>>>>>
>>>>>>> Hi Manula,
>>>>>>>
>>>>>>> You can see those values in the same log I attached in the previous
>>>>>>> mail.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>> rajkum...@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi Chamila,
>>>>>>>>
>>>>>>>> I couldn't find any large values sent by agent. There are traces
>>>>>>>> for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>
>>>>>>>> 1.
>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte <
>>>>>>>> manu...@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Shall we enable the cartridge agent debug logs and see what are
>>>>>>>>> the load average values sent from the agent. If those values are
>>>>>>>>> normal
>>>>>>>>> then we can narrow down to the CEP.
>>>>>>>>>
>>>>>>>>> Thanks !
>>>>>>>>>
>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>> chami...@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Raj,
>>>>>>>>>>
>>>>>>>>>> Can you inspect the CEP trace logs for the same time period? It
>>>>>>>>>> will have the events published from the agent.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Chamila de Alwis
>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>>> rajkum...@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I am doing a test round with M3 with python agent. And faced
>>>>>>>>>>> this very strange scenario, $Subject.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>> average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>> average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>> average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>> average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Avg
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Second
>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain
>>>>>>>>>>> [value]
>>>>>>>>>>> 200000.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>> average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Second
>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>> average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Second
>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>> average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>> average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Grad of
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>> 0.11670045
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>> KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Avg
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>> KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Second
>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain
>>>>>>>>>>> [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>> KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Avg
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>> KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>> average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>> average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Avg
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>> KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Grad of
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>> KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Second
>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain
>>>>>>>>>>> [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>> KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Second
>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value]
>>>>>>>>>>> 2000000.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>> average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>> average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Grad of
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>> KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Avg
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>> KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>> average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>> average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Grad of
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>> KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>> average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Grad of
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>> KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>> average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>> average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>> average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Grad of
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>> KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>> Avg
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>> KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor}
>>>>>>>>>>> -
>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor}
>>>>>>>>>>> -
>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] :
>>>>>>>>>>> php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum
>>>>>>>>>>> rule:
>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete
>>>>>>>>>>> containers
>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check]
>>>>>>>>>>> [cluster]
>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} -
>>>>>>>>>>> Minimum
>>>>>>>>>>> check executed for :
>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor}
>>>>>>>>>>> -
>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag
>>>>>>>>>>> of
>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor}
>>>>>>>>>>> -
>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] :
>>>>>>>>>>> php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} -
>>>>>>>>>>> Predicting the
>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0
>>>>>>>>>>> , [time
>>>>>>>>>>> intervals]: 1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} -
>>>>>>>>>>> Predicting the
>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0
>>>>>>>>>>> , [time
>>>>>>>>>>> intervals]: 1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} -
>>>>>>>>>>> Predicting the
>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value :
>>>>>>>>>>> 28.799999237060547
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up]
>>>>>>>>>>> [cluster] :
>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl}
>>>>>>>>>>> -
>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster :
>>>>>>>>>>> php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl}
>>>>>>>>>>> -
>>>>>>>>>>> Cloud Controller is delegating request to update a replication
>>>>>>>>>>> controller
>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I couldn't investigate the agent side, since containers are
>>>>>>>>>>> destroyed.
>>>>>>>>>>>
>>>>>>>>>>> Any idea?
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>> Software Engineer
>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>
>>>>>>>>> email : manu...@wso2.com / man...@apache.org
>>>>>>>>> phone : +94 772492511
>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Rajkumar Rajaratnam
>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Rajkumar Rajaratnam
>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Manula Chathurika Thantriwatte
>>>>>> Software Engineer
>>>>>> WSO2 Inc. : http://wso2.com
>>>>>> lean . enterprise . middleware
>>>>>>
>>>>>> email : manu...@wso2.com / man...@apache.org
>>>>>> phone : +94 772492511
>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Manula Chathurika Thantriwatte
>>>>> Software Engineer
>>>>> WSO2 Inc. : http://wso2.com
>>>>> lean . enterprise . middleware
>>>>>
>>>>> email : manu...@wso2.com / man...@apache.org
>>>>> phone : +94 772492511
>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Rajkumar Rajaratnam
>>>> Software Engineer | WSO2, Inc.
>>>> Mobile +94777568639 | +94783498120
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Manula Chathurika Thantriwatte
>>> Software Engineer
>>> WSO2 Inc. : http://wso2.com
>>> lean . enterprise . middleware
>>>
>>> email : manu...@wso2.com / man...@apache.org
>>> phone : +94 772492511
>>> blog : http://manulachathurika.blogspot.com/
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Rajkumar Rajaratnam
>> Software Engineer | WSO2, Inc.
>> Mobile +94777568639 | +94783498120
>>
>
>
>
> --
> Regards,
> Manula Chathurika Thantriwatte
> Software Engineer
> WSO2 Inc. : http://wso2.com
> lean . enterprise . middleware
>
> email : manu...@wso2.com / man...@apache.org
> phone : +94 772492511
> blog : http://manulachathurika.blogspot.com/
>
>
>
>
>
>
> --
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
> Blog: http://nirmalfdo.blogspot.com/
>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
