Hi,
On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <im...@apache.org> wrote:
> Hi Nirmal,
>
> I do not think the milliseconds to seconds convertion is correct here.
>
> As I see we are taking the difference of two timestamp values and then
> dividing it by 1000. The corect way might be to first divide each value by
> 1000 and then take the difference.
>
No Imesh. There are no difference between *taking the the difference and
dividing it by 1000* and *dividing it by 1000 and taking the difference*.
> On the other hand we might not need to convert these values to seconds
> since we are taking a time difference and calculating a gradient.
>
> I did a quick test with the following sample:
> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val: 12.0
> Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>
> *According previous code:*
> long tGap = t2 - t1;
> double gradient = 0.0;
> if (tGap > 0) {
> gradient = ((lastVal - firstVal) * 1000) / tGap;
> }
>
> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
> gradient: -1000.0
>
> *According to your fix:*
> long millisecondsForASecond = 1000;
> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
> millisecondsForASecond;
> double gradient = 0.0;
> if (tGap > 0) {
> gradient = ((lastVal - firstVal) * millisecondsForASecond) /
> tGap;
> }
>
> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
> gradient: -5.0
>
> *According to an online gradient calculator:*
> gradient: -1
>
> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>
> According to the online gradient calculator (assuming their calculation is
> correct), the calculation in your fix is not correct. I believe the logic
> should be simple as follows:
>
> long tGap = t2 - t1;
> double gradient = 0.0;
> if (tGap > 0) {
> gradient = ((lastVal - firstVal)) / tGap;
> }
>
> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
> gradient: -1.0
>
And AFAIK, standard way to calculate gradient = (v1-v2)/timeInSeconds.
Well, we can divide it by what ever time units we wants, but the result's
unit will be different.
In Nirmal's case, he get will get *stats/seconds*. In your case, you will
get *stats/milliseconds*
And AS prediction is based on *seconds*, CEP should use *seconds* for the
calculation.
wdyt?
Thanks.
> Thanks
>
>
> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <nirmal070...@gmail.com>
> wrote:
>
>> Hi Guys,
>>
>> So, I got a chance to reproduce and analyze this issue.
>>
>> How we calculate gradient of two events?
>>
>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>
>> tx - time in *milliseconds* when xth event occurred
>> vx - value (memory, cpu etc.) that xth event carries
>>
>> *time gap = t(2-1) = t2 -t1 milliseconds*
>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>
>> Hence,
>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 - t1)*
>>
>> I've enabled debug logs for CEP extension;
>>
>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>
>> Please find the following 3 logs extracted from the debug logs;
>>
>> ===================================================================
>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time Gap:
>> 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>>
>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>> Gradient:
>> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time
>> Gap: 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>>
>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>> 1415213232154 hash: 155426542
>> ===================================================================
>>
>> So, as you can see the reason behind a large value is when the time gap
>> between two subjected events is less than 1 second. This could happen since
>> events are coming from different asynchronous agents and also when there
>> are less number of events.
>>
>> So, the fix I propose is a very simple one and it will not compromise
>> anything AFAIS.
>>
>> Fix is to calculate time gap as follows;
>> *____ t2 - t1*
>> *| yes?*
>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>> *|____ 1000*
>>
>> I have tested this and works fine.
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Manula Chathurika Thantriwatte <manu...@wso2.com>
>> Date: Tue, Oct 28, 2014 at 5:16 AM
>> Subject: Re: CEP sends very large values for gradient and second
>> derivative of load average
>> To: "dev@stratos.apache.org" <dev@stratos.apache.org>
>>
>>
>> Hi Raj,
>>
>> Yes it has. But it's very easy to find the values from the agent with the
>> time stamp. Then we can search that time stamp in the CEP trace logs and
>> compare the summarized values.
>>
>> Thanks !
>>
>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <rajkum...@wso2.com>
>> wrote:
>>
>>>
>>>
>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <
>>> manu...@wso2.com> wrote:
>>>
>>>> Hi Raj,
>>>>
>>>> Shall we enable the cartridge agent debug logs. Then we can compare
>>>> both CEP and agent logs with the time stamp. Then we can narrow down where
>>>> the actual problem is.
>>>>
>>>
>>> Manula, cep-trace log have the values sent by cartridge agent also
>>> right?
>>>
>>>
>>>>
>>>> Thanks !
>>>>
>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>> rajkum...@wso2.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
>>>>> manu...@wso2.com> wrote:
>>>>>
>>>>>> Hi Raj,
>>>>>>
>>>>>> Is this happen frequently or intermittently ? Once I go though the
>>>>>> logs I'm unable to find the high input values for load average. But
>>>>>> output
>>>>>> have high values.
>>>>>>
>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>> JMSOutputAdaptor, sent
>>>>>>
>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>
>>>>>
>>>>> This is happening intermittently. Experienced now too. This is a
>>>>> serious issue. We need to somehow sort this out.
>>>>>
>>>>>
>>>>>>
>>>>>> Also networkPartitionId is empty. In the current execution plans it
>>>>>> took networkPartitionId for the calculation. IMO we need to update the
>>>>>> CEP
>>>>>> execution plans for 4.1.0.
>>>>>>
>>>>>> Thanks !
>>>>>>
>>>>>>
>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte <
>>>>>> manu...@wso2.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>
>>>>>>> Thanks !
>>>>>>>
>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>> rajkum...@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi Manula,
>>>>>>>>
>>>>>>>> You can see those values in the same log I attached in the previous
>>>>>>>> mail.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>> rajkum...@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Chamila,
>>>>>>>>>
>>>>>>>>> I couldn't find any large values sent by agent. There are traces
>>>>>>>>> for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>
>>>>>>>>> 1.
>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte <
>>>>>>>>> manu...@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Shall we enable the cartridge agent debug logs and see what are
>>>>>>>>>> the load average values sent from the agent. If those values are
>>>>>>>>>> normal
>>>>>>>>>> then we can narrow down to the CEP.
>>>>>>>>>>
>>>>>>>>>> Thanks !
>>>>>>>>>>
>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>> chami...@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>
>>>>>>>>>>> Can you inspect the CEP trace logs for the same time period? It
>>>>>>>>>>> will have the events published from the agent.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>>>> rajkum...@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I am doing a test round with M3 with python agent. And faced
>>>>>>>>>>>> this very strange scenario, $Subject.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>> 28.718182
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Second
>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain
>>>>>>>>>>>> [value]
>>>>>>>>>>>> 200000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Second
>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Second
>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>> 0.11670045
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Second
>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain
>>>>>>>>>>>> [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Second
>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain
>>>>>>>>>>>> [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Second
>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value]
>>>>>>>>>>>> 2000000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>> 28.738462
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor}
>>>>>>>>>>>> -
>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor}
>>>>>>>>>>>> -
>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] :
>>>>>>>>>>>> php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum
>>>>>>>>>>>> rule:
>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete
>>>>>>>>>>>> containers
>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check]
>>>>>>>>>>>> [cluster]
>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} -
>>>>>>>>>>>> Minimum
>>>>>>>>>>>> check executed for :
>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor}
>>>>>>>>>>>> -
>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true
>>>>>>>>>>>> flag of
>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor}
>>>>>>>>>>>> -
>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] :
>>>>>>>>>>>> php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling
>>>>>>>>>>>> rule
>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} -
>>>>>>>>>>>> Predicting the
>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0
>>>>>>>>>>>> , [time
>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} -
>>>>>>>>>>>> Predicting the
>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]:
>>>>>>>>>>>> 0.0 , [time
>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} -
>>>>>>>>>>>> Predicting the
>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second
>>>>>>>>>>>> derivative]:
>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value :
>>>>>>>>>>>> 28.799999237060547
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided
>>>>>>>>>>>> to
>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl}
>>>>>>>>>>>> -
>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster :
>>>>>>>>>>>> php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl}
>>>>>>>>>>>> -
>>>>>>>>>>>> Cloud Controller is delegating request to update a replication
>>>>>>>>>>>> controller
>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I couldn't investigate the agent side, since containers are
>>>>>>>>>>>> destroyed.
>>>>>>>>>>>>
>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>> Software Engineer
>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>
>>>>>>>>>> email : manu...@wso2.com / man...@apache.org
>>>>>>>>>> phone : +94 772492511
>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Rajkumar Rajaratnam
>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Manula Chathurika Thantriwatte
>>>>>>> Software Engineer
>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>> lean . enterprise . middleware
>>>>>>>
>>>>>>> email : manu...@wso2.com / man...@apache.org
>>>>>>> phone : +94 772492511
>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Manula Chathurika Thantriwatte
>>>>>> Software Engineer
>>>>>> WSO2 Inc. : http://wso2.com
>>>>>> lean . enterprise . middleware
>>>>>>
>>>>>> email : manu...@wso2.com / man...@apache.org
>>>>>> phone : +94 772492511
>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Rajkumar Rajaratnam
>>>>> Software Engineer | WSO2, Inc.
>>>>> Mobile +94777568639 | +94783498120
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Manula Chathurika Thantriwatte
>>>> Software Engineer
>>>> WSO2 Inc. : http://wso2.com
>>>> lean . enterprise . middleware
>>>>
>>>> email : manu...@wso2.com / man...@apache.org
>>>> phone : +94 772492511
>>>> blog : http://manulachathurika.blogspot.com/
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Rajkumar Rajaratnam
>>> Software Engineer | WSO2, Inc.
>>> Mobile +94777568639 | +94783498120
>>>
>>
>>
>>
>> --
>> Regards,
>> Manula Chathurika Thantriwatte
>> Software Engineer
>> WSO2 Inc. : http://wso2.com
>> lean . enterprise . middleware
>>
>> email : manu...@wso2.com / man...@apache.org
>> phone : +94 772492511
>> blog : http://manulachathurika.blogspot.com/
>>
>>
>>
>>
>>
>>
>> --
>> Best Regards,
>> Nirmal
>>
>> Nirmal Fernando.
>> PPMC Member & Committer of Apache Stratos,
>> Senior Software Engineer, WSO2 Inc.
>>
>> Blog: http://nirmalfdo.blogspot.com/
>>
>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
--
Raj
