On Thu, Nov 6, 2014 at 4:33 AM, Imesh Gunaratne <im...@apache.org> wrote:
> Hi Nirmal,
>
> I do not think the milliseconds to seconds convertion is correct here.
>
> As I see we are taking the difference of two timestamp values and then
> dividing it by 1000. The corect way might be to first divide each value by
> 1000 and then take the difference.
>
(5-1)/2 = 5/2 - 1/2 is the Math I know :-)
>
> On the other hand we might not need to convert these values to seconds
> since we are taking a time difference and calculating a gradient.
>
It's a time difference, so the value still matters, if it's division of
same units, then it can be neglected.
>
> I did a quick test with the following sample:
> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val: 12.0
> Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>
> *According previous code:*
> long tGap = t2 - t1;
> double gradient = 0.0;
> if (tGap > 0) {
> gradient = ((lastVal - firstVal) * 1000) / tGap;
> }
>
> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
> gradient: -1000.0
>
> *According to your fix:*
> long millisecondsForASecond = 1000;
> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
> millisecondsForASecond;
> double gradient = 0.0;
> if (tGap > 0) {
> gradient = ((lastVal - firstVal) * millisecondsForASecond) /
> tGap;
> }
>
> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
> gradient: -5.0
>
> *According to an online gradient calculator:*
> gradient: -1
>
> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>
> According to the online gradient calculator (assuming their calculation is
> correct), the calculation in your fix is not correct. I believe the logic
> should be simple as follows:
>
> long tGap = t2 - t1;
> double gradient = 0.0;
> if (tGap > 0) {
> gradient = ((lastVal - firstVal)) / tGap;
> }
>
> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
> gradient: -1.0
>
This is not per second value.
So, what I've done here is to put a reasonable constraint on the events
saying, two events should be at least 1s apart from each other before doing
gradient (per second) calculation.
And this is the tangent value we're talking about and tangent can span from
0 to (+/-)infinity.
>
> Thanks
>
>
> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <nirmal070...@gmail.com>
> wrote:
>
>> Hi Guys,
>>
>> So, I got a chance to reproduce and analyze this issue.
>>
>> How we calculate gradient of two events?
>>
>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>
>> tx - time in *milliseconds* when xth event occurred
>> vx - value (memory, cpu etc.) that xth event carries
>>
>> *time gap = t(2-1) = t2 -t1 milliseconds*
>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>
>> Hence,
>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 - t1)*
>>
>> I've enabled debug logs for CEP extension;
>>
>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>
>> Please find the following 3 logs extracted from the debug logs;
>>
>> ===================================================================
>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time Gap:
>> 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>>
>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>> Gradient:
>> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time
>> Gap: 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>>
>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>> 1415213232154 hash: 155426542
>> ===================================================================
>>
>> So, as you can see the reason behind a large value is when the time gap
>> between two subjected events is less than 1 second. This could happen since
>> events are coming from different asynchronous agents and also when there
>> are less number of events.
>>
>> So, the fix I propose is a very simple one and it will not compromise
>> anything AFAIS.
>>
>> Fix is to calculate time gap as follows;
>> *____ t2 - t1*
>> *| yes?*
>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>> *|____ 1000*
>>
>> I have tested this and works fine.
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Manula Chathurika Thantriwatte <manu...@wso2.com>
>> Date: Tue, Oct 28, 2014 at 5:16 AM
>> Subject: Re: CEP sends very large values for gradient and second
>> derivative of load average
>> To: "dev@stratos.apache.org" <dev@stratos.apache.org>
>>
>>
>> Hi Raj,
>>
>> Yes it has. But it's very easy to find the values from the agent with the
>> time stamp. Then we can search that time stamp in the CEP trace logs and
>> compare the summarized values.
>>
>> Thanks !
>>
>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <rajkum...@wso2.com>
>> wrote:
>>
>>>
>>>
>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <
>>> manu...@wso2.com> wrote:
>>>
>>>> Hi Raj,
>>>>
>>>> Shall we enable the cartridge agent debug logs. Then we can compare
>>>> both CEP and agent logs with the time stamp. Then we can narrow down where
>>>> the actual problem is.
>>>>
>>>
>>> Manula, cep-trace log have the values sent by cartridge agent also
>>> right?
>>>
>>>
>>>>
>>>> Thanks !
>>>>
>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>> rajkum...@wso2.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
>>>>> manu...@wso2.com> wrote:
>>>>>
>>>>>> Hi Raj,
>>>>>>
>>>>>> Is this happen frequently or intermittently ? Once I go though the
>>>>>> logs I'm unable to find the high input values for load average. But
>>>>>> output
>>>>>> have high values.
>>>>>>
>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>> JMSOutputAdaptor, sent
>>>>>>
>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>
>>>>>
>>>>> This is happening intermittently. Experienced now too. This is a
>>>>> serious issue. We need to somehow sort this out.
>>>>>
>>>>>
>>>>>>
>>>>>> Also networkPartitionId is empty. In the current execution plans it
>>>>>> took networkPartitionId for the calculation. IMO we need to update the
>>>>>> CEP
>>>>>> execution plans for 4.1.0.
>>>>>>
>>>>>> Thanks !
>>>>>>
>>>>>>
>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte <
>>>>>> manu...@wso2.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>
>>>>>>> Thanks !
>>>>>>>
>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>> rajkum...@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi Manula,
>>>>>>>>
>>>>>>>> You can see those values in the same log I attached in the previous
>>>>>>>> mail.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>> rajkum...@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Chamila,
>>>>>>>>>
>>>>>>>>> I couldn't find any large values sent by agent. There are traces
>>>>>>>>> for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>
>>>>>>>>> 1.
>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte <
>>>>>>>>> manu...@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Shall we enable the cartridge agent debug logs and see what are
>>>>>>>>>> the load average values sent from the agent. If those values are
>>>>>>>>>> normal
>>>>>>>>>> then we can narrow down to the CEP.
>>>>>>>>>>
>>>>>>>>>> Thanks !
>>>>>>>>>>
>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>> chami...@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>
>>>>>>>>>>> Can you inspect the CEP trace logs for the same time period? It
>>>>>>>>>>> will have the events published from the agent.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>>>> rajkum...@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I am doing a test round with M3 with python agent. And faced
>>>>>>>>>>>> this very strange scenario, $Subject.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>> 28.718182
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Second
>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain
>>>>>>>>>>>> [value]
>>>>>>>>>>>> 200000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Second
>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Second
>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>> 0.11670045
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Second
>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain
>>>>>>>>>>>> [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Second
>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain
>>>>>>>>>>>> [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Second
>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value]
>>>>>>>>>>>> 2000000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>> 28.738462
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>> average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>> Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster]
>>>>>>>>>>>> KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor}
>>>>>>>>>>>> -
>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor}
>>>>>>>>>>>> -
>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] :
>>>>>>>>>>>> php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum
>>>>>>>>>>>> rule:
>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete
>>>>>>>>>>>> containers
>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check]
>>>>>>>>>>>> [cluster]
>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} -
>>>>>>>>>>>> Minimum
>>>>>>>>>>>> check executed for :
>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor}
>>>>>>>>>>>> -
>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true
>>>>>>>>>>>> flag of
>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor}
>>>>>>>>>>>> -
>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] :
>>>>>>>>>>>> php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling
>>>>>>>>>>>> rule
>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} -
>>>>>>>>>>>> Predicting the
>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0
>>>>>>>>>>>> , [time
>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} -
>>>>>>>>>>>> Predicting the
>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]:
>>>>>>>>>>>> 0.0 , [time
>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} -
>>>>>>>>>>>> Predicting the
>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second
>>>>>>>>>>>> derivative]:
>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value :
>>>>>>>>>>>> 28.799999237060547
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided
>>>>>>>>>>>> to
>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up]
>>>>>>>>>>>> [cluster] :
>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl}
>>>>>>>>>>>> -
>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster :
>>>>>>>>>>>> php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl}
>>>>>>>>>>>> -
>>>>>>>>>>>> Cloud Controller is delegating request to update a replication
>>>>>>>>>>>> controller
>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I couldn't investigate the agent side, since containers are
>>>>>>>>>>>> destroyed.
>>>>>>>>>>>>
>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>> Software Engineer
>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>
>>>>>>>>>> email : manu...@wso2.com / man...@apache.org
>>>>>>>>>> phone : +94 772492511
>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Rajkumar Rajaratnam
>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Manula Chathurika Thantriwatte
>>>>>>> Software Engineer
>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>> lean . enterprise . middleware
>>>>>>>
>>>>>>> email : manu...@wso2.com / man...@apache.org
>>>>>>> phone : +94 772492511
>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Manula Chathurika Thantriwatte
>>>>>> Software Engineer
>>>>>> WSO2 Inc. : http://wso2.com
>>>>>> lean . enterprise . middleware
>>>>>>
>>>>>> email : manu...@wso2.com / man...@apache.org
>>>>>> phone : +94 772492511
>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Rajkumar Rajaratnam
>>>>> Software Engineer | WSO2, Inc.
>>>>> Mobile +94777568639 | +94783498120
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Manula Chathurika Thantriwatte
>>>> Software Engineer
>>>> WSO2 Inc. : http://wso2.com
>>>> lean . enterprise . middleware
>>>>
>>>> email : manu...@wso2.com / man...@apache.org
>>>> phone : +94 772492511
>>>> blog : http://manulachathurika.blogspot.com/
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Rajkumar Rajaratnam
>>> Software Engineer | WSO2, Inc.
>>> Mobile +94777568639 | +94783498120
>>>
>>
>>
>>
>> --
>> Regards,
>> Manula Chathurika Thantriwatte
>> Software Engineer
>> WSO2 Inc. : http://wso2.com
>> lean . enterprise . middleware
>>
>> email : manu...@wso2.com / man...@apache.org
>> phone : +94 772492511
>> blog : http://manulachathurika.blogspot.com/
>>
>>
>>
>>
>>
>>
>> --
>> Best Regards,
>> Nirmal
>>
>> Nirmal Fernando.
>> PPMC Member & Committer of Apache Stratos,
>> Senior Software Engineer, WSO2 Inc.
>>
>> Blog: http://nirmalfdo.blogspot.com/
>>
>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
