On Thu, Nov 6, 2014 at 7:15 PM, Lahiru Sandaruwan <lahi...@wso2.com> wrote:
> Hi all, > > Very interesting discussion and thanks Nirmal for initiating this! > > I also did some research on this. > I think best approach to find gradient and second derivative is to "curve > fitting". Refer "Locally weighted linear regression" section at [1]. We > should be calculating gradient and second derivative by differentiating the > equation of fitted line. > > We would need a second degree polynomial fitter for this, where we can use > Apache commons math library for this. Refer the sample at [2], we need to > run this with degree 2 instead of 3. > > E.g. > So if get degree 2 polynomial fitter, we will have an equation like below > where value(v) is our statistic value, time(t) is the time of event, > gradient(g), and second derivative(sd). > > Equation we get from received events, > v = a*t*t + b*t + c > > g = 2*a*t > sd = 2*a > > We can get *g* by applying the median of the time stamp sample. > > I have run a sample successfully. I can complete the work on CEP extension > if you guys agree. > > In addition to this, we should also restrict the sample size as Akila > suggested. Ideally we should receive 6 events per minute for memberwise > stats and "6 * no. of instances" for clusterwise stats. We should put a > constraint with this much of percentage(say 80%) should be there to > calculate *g *and* sd.* > > Also we should not calculated gradient of stats sent from different > members in clusterwise scenario as Imesh mentioned. That causes this type > of small time gap issues. > I have thought of a solution for this. Also most importantly the solution i propose suits very well for the prediction requirement at Autoscaler, than what we have now. Please go through and give feedback. *CEP's task* - Find memberwise curves that fits events received in 1 minute(or required) window at CEP - We send the parameters of fitted line(a, b, and c in above equation) with the timestamp of last event(T) in the window, to Autoscaler *Autoscaler's task* - Autoscaler use *v = a*t*t + b*t + c *function to predict the value in any timestamp from the last timestamp E.g. Say we need to find the value(v) after 1 minute(assuming we carried all the calculations in milliseconds), v = a * (T+60000) * (T+60000) + b * (T+60000) + c - So we have memberwise predictions and we can find clusterwise prediction by averaging all the memberwise values. Please send your thoughts. Thanks. > Thanks. > > [1] http://cs229.stanford.edu/notes/cs229-notes1.pdf > [2] http://commons.apache.org/proper/commons-math/userguide/fitting.html > > On Thu, Nov 6, 2014 at 6:41 PM, Imesh Gunaratne <im...@apache.org> wrote: > >> Raj: Thanks for the clarification, it was my misunderstanding. >> >> I did further analysis on this issue, seems like the root cause of this >> problem resides somewhere else. >> >> Currently we have set the cartridge agent's statistics publishing >> interval to 15 seconds. If so how do we receive series of events within a >> second? Seems like we are considering statistics events coming from >> different members for calculating the gradient. This might not be correct, >> if we were to draw a curve of a data set and consider the gradient of those >> values, those values may have come from the same source. Otherwise the >> gradient value we are calculating might not be correct. >> >> Thanks >> >> On Thu, Nov 6, 2014 at 2:05 PM, Udara Liyanage <ud...@wso2.com> wrote: >> >>> Hi, >>> >>> Imagine we put the values in a graph (value against time). The gradient >>> is the slope of the line. When we draw the line via two very near dots, we >>> get a high slope/gradient even for a low increase of the values. This >>> happens when we have only events which are occurred in very near time >>> intervals. >>> >>> On the other hands, when events are distributed among long time period >>> (are more counts in sample) we don't get this high gradient since the line >>> goes via higher dots count where we get a reasonable value for gradient. >>> >>> As I understood the scenario, it is reasonable to make time gap equals >>> to millisInSeconds in case time gap is very small. >>> >>> Touched, not typed. Erroneous words are a feature, not a typo. >>> I'd rather keep it simple Akila (this would complex the scenario of >>> second derivative too) :-) Isn't it? >>> >>> On Thu, Nov 6, 2014 at 9:03 AM, Akila Ravihansa Perera < >>> raviha...@wso2.com> wrote: >>> >>>> Until t2 - t1 > 1 >>>> >>>> On Thu, Nov 6, 2014 at 12:50 AM, Nirmal Fernando < >>>> nirmal070...@gmail.com> wrote: >>>> >>>>> >>>>> >>>>> On Thu, Nov 6, 2014 at 8:33 AM, Akila Ravihansa Perera < >>>>> raviha...@wso2.com> wrote: >>>>> >>>>>> Thanks for the explanation, Nirmal. It makes sense now. >>>>>> >>>>>> Perhaps as an alternative solution, window processor can be set to >>>>>> wait until sufficient events have arrived to calculate the gradient? >>>>>> >>>>> >>>>> What would be the sufficient value? Theoretically in gradient case >>>>> it's 2, in second derivative case it's 3. And we're doing that already. >>>>> >>>>>> >>>>>> On Wed, Nov 5, 2014 at 11:48 PM, Rajkumar Rajaratnam < >>>>>> rajkum...@wso2.com> wrote: >>>>>> >>>>>>> In other words, fix reduces the risks which are coming from bad >>>>>>> samples. >>>>>>> >>>>>>> On Thu, Nov 6, 2014 at 12:08 PM, Rajkumar Rajaratnam < >>>>>>> rajkum...@wso2.com> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne <im...@apache.org> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam < >>>>>>>>> rajkum...@wso2.com> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> No Imesh. There are no difference between *taking the the >>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000 and >>>>>>>>>> taking the difference*. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Raj: I agree, in this scenario it makes no difference, however >>>>>>>>> when we are applying a fomular the unit conversion should happen >>>>>>>>> prior to >>>>>>>>> applying values. >>>>>>>>> >>>>>>>> >>>>>>>> Not exactly Imesh. We can do operations if operands are in the same >>>>>>>> units. So we can do subtract operation in milliseconds. And then we >>>>>>>> can do >>>>>>>> division operation to make it in seconds. On the other hand, we can do >>>>>>>> a >>>>>>>> division operation to make each operands to seconds, then since both >>>>>>>> are >>>>>>>> seconds, we can do a subtract operation. So we can apply the formula if >>>>>>>> operands are in same unit. I guess this is same for all scenarios. >>>>>>>> >>>>>>>> >>>>>>>>> Still I see a problem with this correction, take the same example: >>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First >>>>>>>>> val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157 >>>>>>>>> >>>>>>>>> *Manual calulation:* >>>>>>>>> The value difference: 7 - 12 = -5 >>>>>>>>> Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000 = >>>>>>>>> 0.005 sec >>>>>>>>> *Gradient = - 5/0.005 = -1000* >>>>>>>>> >>>>>>>>> *According to the current fix:* >>>>>>>>> long millisecondsForASecond = 1000; >>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 : >>>>>>>>> millisecondsForASecond; >>>>>>>>> double gradient = 0.0; >>>>>>>>> if (tGap > 0) { >>>>>>>>> gradient = ((lastVal - firstVal) * >>>>>>>>> millisecondsForASecond) / tGap; >>>>>>>>> } >>>>>>>>> >>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7 >>>>>>>>> Gradient = -5 * 1000 / 1000 = - 5 >>>>>>>>> >>>>>>>> >>>>>>>> IMO, this is a sufficient solution. >>>>>>>> >>>>>>>> Let me explain how I see Nirmal's solution. >>>>>>>> >>>>>>>> We are taking only first and last values in a window for gradient >>>>>>>> calculation. If there are more events in that window, and if that is a >>>>>>>> good >>>>>>>> sample, tGap will be taking some higher values than 1s. This is usual >>>>>>>> situation. Nirmal's fix is nothing to do with this scenario. >>>>>>>> >>>>>>>> If there are few events in that window(say only 2), they can be >>>>>>>> really nearer. So there is a possibility that tGap can take a value >>>>>>>> near to >>>>>>>> 0. In that case gradient will be a very big value. Hence, if the tGap >>>>>>>> is >>>>>>>> very low (less than 1 second), then we can assume that the first and >>>>>>>> last >>>>>>>> events actually occurred in 1 second gap. The reason why tGap is less >>>>>>>> than >>>>>>>> 0 is, there are only few events in that window, and that sample is not >>>>>>>> good >>>>>>>> enough. But this a rare situation, it can only happen once in a while. >>>>>>>> Nirmal's fix is only dealing with this scenario. Fix imposes a lower >>>>>>>> limit >>>>>>>> in the time interval between two events. This fix is really important >>>>>>>> to >>>>>>>> deal with samples which are not good enough. >>>>>>>> >>>>>>>> By previous formula, we get -1000. This happened because the sample >>>>>>>> is not good enough (may be there are few events). Fix makes it to -5. >>>>>>>> >>>>>>>> So my understanding is that the solution is good enough for now. >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>>> >>>>>>>>> Nirmal: Can you please justify this? >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando < >>>>>>>>> nirmal070...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <gay...@wso2.com >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> AFAIU this is not related with the gradient calculation formula >>>>>>>>>>> that we used. But It is related with the how we calculate gradient >>>>>>>>>>> when the >>>>>>>>>>> denominator become a small number.IMO when the divisor >>>>>>>>>>> (timeInSeconds) >>>>>>>>>>> approaches to the small number we do need to normalize the result. >>>>>>>>>>> >>>>>>>>>>> IMO we do have max and min scale up and down values. So based on >>>>>>>>>>> that we can calculate the gradient on that value range.Simply if the >>>>>>>>>>> denominator approaches zero result approaches to infinity. >>>>>>>>>>> >>>>>>>>>>> I think that's what Nirmal doing here by making the lower >>>>>>>>>>> threshold as "millisecondsForASecond" >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> You got it! thanks. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 : >>>>>>>>>>> millisecondsForASecond; >>>>>>>>>>> double gradient = 0.0; >>>>>>>>>>> if (tGap > 0) { >>>>>>>>>>> gradient = ((lastVal - firstVal) * >>>>>>>>>>> millisecondsForASecond) / tGap; >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Gayan >>>>>>>>>>> >>>>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam < >>>>>>>>>>> rajkum...@wso2.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne < >>>>>>>>>>>> im...@apache.org> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Nirmal, >>>>>>>>>>>>> >>>>>>>>>>>>> I do not think the milliseconds to seconds convertion is >>>>>>>>>>>>> correct here. >>>>>>>>>>>>> >>>>>>>>>>>>> As I see we are taking the difference of two timestamp values >>>>>>>>>>>>> and then dividing it by 1000. The corect way might be to first >>>>>>>>>>>>> divide each >>>>>>>>>>>>> value by 1000 and then take the difference. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> No Imesh. There are no difference between *taking the the >>>>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000 >>>>>>>>>>>> and taking the difference*. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On the other hand we might not need to convert these values to >>>>>>>>>>>>> seconds since we are taking a time difference and calculating a >>>>>>>>>>>>> gradient. >>>>>>>>>>>>> >>>>>>>>>>>>> I did a quick test with the following sample: >>>>>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First >>>>>>>>>>>>> val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157 >>>>>>>>>>>>> >>>>>>>>>>>>> *According previous code:* >>>>>>>>>>>>> long tGap = t2 - t1; >>>>>>>>>>>>> double gradient = 0.0; >>>>>>>>>>>>> if (tGap > 0) { >>>>>>>>>>>>> gradient = ((lastVal - firstVal) * 1000) / tGap; >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7 >>>>>>>>>>>>> gradient: -1000.0 >>>>>>>>>>>>> >>>>>>>>>>>>> *According to your fix:* >>>>>>>>>>>>> long millisecondsForASecond = 1000; >>>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 >>>>>>>>>>>>> : millisecondsForASecond; >>>>>>>>>>>>> double gradient = 0.0; >>>>>>>>>>>>> if (tGap > 0) { >>>>>>>>>>>>> gradient = ((lastVal - firstVal) * >>>>>>>>>>>>> millisecondsForASecond) / tGap; >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7 >>>>>>>>>>>>> gradient: -5.0 >>>>>>>>>>>>> >>>>>>>>>>>>> *According to an online gradient calculator:* >>>>>>>>>>>>> gradient: -1 >>>>>>>>>>>>> >>>>>>>>>>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19 >>>>>>>>>>>>> >>>>>>>>>>>>> According to the online gradient calculator (assuming their >>>>>>>>>>>>> calculation is correct), the calculation in your fix is not >>>>>>>>>>>>> correct. I >>>>>>>>>>>>> believe the logic should be simple as follows: >>>>>>>>>>>>> >>>>>>>>>>>>> long tGap = t2 - t1; >>>>>>>>>>>>> double gradient = 0.0; >>>>>>>>>>>>> if (tGap > 0) { >>>>>>>>>>>>> gradient = ((lastVal - firstVal)) / tGap; >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7 >>>>>>>>>>>>> gradient: -1.0 >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> And AFAIK, standard way to calculate gradient = >>>>>>>>>>>> (v1-v2)/timeInSeconds. >>>>>>>>>>>> >>>>>>>>>>>> Well, we can divide it by what ever time units we wants, but >>>>>>>>>>>> the result's unit will be different. >>>>>>>>>>>> >>>>>>>>>>>> In Nirmal's case, he get will get *stats/seconds*. In your >>>>>>>>>>>> case, you will get *stats/milliseconds* >>>>>>>>>>>> >>>>>>>>>>>> And AS prediction is based on *seconds*, CEP should use >>>>>>>>>>>> *seconds* for the calculation. >>>>>>>>>>>> >>>>>>>>>>>> wdyt? >>>>>>>>>>>> >>>>>>>>>>>> Thanks. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando < >>>>>>>>>>>>> nirmal070...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Guys, >>>>>>>>>>>>>> >>>>>>>>>>>>>> So, I got a chance to reproduce and analyze this issue. >>>>>>>>>>>>>> >>>>>>>>>>>>>> How we calculate gradient of two events? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Say the events are; e1(t1,v1) and e2(t2,v2) >>>>>>>>>>>>>> >>>>>>>>>>>>>> tx - time in *milliseconds* when xth event occurred >>>>>>>>>>>>>> vx - value (memory, cpu etc.) that xth event carries >>>>>>>>>>>>>> >>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 milliseconds* >>>>>>>>>>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hence, >>>>>>>>>>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 >>>>>>>>>>>>>> - t1)* >>>>>>>>>>>>>> >>>>>>>>>>>>>> I've enabled debug logs for CEP extension; >>>>>>>>>>>>>> >>>>>>>>>>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG >>>>>>>>>>>>>> >>>>>>>>>>>>>> Please find the following 3 logs extracted from the debug >>>>>>>>>>>>>> logs; >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> =================================================================== >>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG >>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} >>>>>>>>>>>>>> - >>>>>>>>>>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 >>>>>>>>>>>>>> *Time >>>>>>>>>>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash: >>>>>>>>>>>>>> 155426542 >>>>>>>>>>>>>> >>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG >>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} >>>>>>>>>>>>>> - Gradient: >>>>>>>>>>>>>> -999.9999999999998 Last val: 7.000000000000001 First val: >>>>>>>>>>>>>> 12.0 *Time Gap: 5* t1: 1415213232152 t2: 1415213232157 hash: >>>>>>>>>>>>>> 155426542 >>>>>>>>>>>>>> >>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG >>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} >>>>>>>>>>>>>> - >>>>>>>>>>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First >>>>>>>>>>>>>> val: >>>>>>>>>>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2: >>>>>>>>>>>>>> 1415213232154 hash: 155426542 >>>>>>>>>>>>>> >>>>>>>>>>>>>> =================================================================== >>>>>>>>>>>>>> >>>>>>>>>>>>>> So, as you can see the reason behind a large value is when >>>>>>>>>>>>>> the time gap between two subjected events is less than 1 second. >>>>>>>>>>>>>> This could >>>>>>>>>>>>>> happen since events are coming from different asynchronous >>>>>>>>>>>>>> agents and also >>>>>>>>>>>>>> when there are less number of events. >>>>>>>>>>>>>> >>>>>>>>>>>>>> So, the fix I propose is a very simple one and it will not >>>>>>>>>>>>>> compromise anything AFAIS. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Fix is to calculate time gap as follows; >>>>>>>>>>>>>> *____ >>>>>>>>>>>>>> t2 - t1* >>>>>>>>>>>>>> *| >>>>>>>>>>>>>> yes?* >>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---* >>>>>>>>>>>>>> *|____ >>>>>>>>>>>>>> 1000* >>>>>>>>>>>>>> >>>>>>>>>>>>>> I have tested this and works fine. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ---------- Forwarded message ---------- >>>>>>>>>>>>>> From: Manula Chathurika Thantriwatte <manu...@wso2.com> >>>>>>>>>>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM >>>>>>>>>>>>>> Subject: Re: CEP sends very large values for gradient and >>>>>>>>>>>>>> second derivative of load average >>>>>>>>>>>>>> To: "dev@stratos.apache.org" <dev@stratos.apache.org> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Raj, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Yes it has. But it's very easy to find the values from the >>>>>>>>>>>>>> agent with the time stamp. Then we can search that time stamp in >>>>>>>>>>>>>> the CEP >>>>>>>>>>>>>> trace logs and compare the summarized values. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks ! >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam < >>>>>>>>>>>>>> rajkum...@wso2.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika >>>>>>>>>>>>>>> Thantriwatte <manu...@wso2.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Raj, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs. Then we can >>>>>>>>>>>>>>>> compare both CEP and agent logs with the time stamp. Then we >>>>>>>>>>>>>>>> can narrow >>>>>>>>>>>>>>>> down where the actual problem is. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Manula, cep-trace log have the values sent by cartridge >>>>>>>>>>>>>>> agent also right? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks ! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam < >>>>>>>>>>>>>>>> rajkum...@wso2.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika >>>>>>>>>>>>>>>>> Thantriwatte <manu...@wso2.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Raj, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Is this happen frequently or intermittently ? Once I go >>>>>>>>>>>>>>>>>> though the logs I'm unable to find the high input values for >>>>>>>>>>>>>>>>>> load average. >>>>>>>>>>>>>>>>>> But output have high values. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] >>>>>>>>>>>>>>>>>> INFO EVENT_TRACE_LOGGER TenantId=-1234 : Output Event >>>>>>>>>>>>>>>>>> Adaptor : >>>>>>>>>>>>>>>>>> JMSOutputAdaptor, sent >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}} >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> This is happening intermittently. Experienced now too. >>>>>>>>>>>>>>>>> This is a serious issue. We need to somehow sort this out. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Also networkPartitionId is empty. In the current >>>>>>>>>>>>>>>>>> execution plans it took networkPartitionId for the >>>>>>>>>>>>>>>>>> calculation. IMO we need >>>>>>>>>>>>>>>>>> to update the CEP execution plans for 4.1.0. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks ! >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika >>>>>>>>>>>>>>>>>> Thantriwatte <manu...@wso2.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks Raj. I'll go through them. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks ! >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam < >>>>>>>>>>>>>>>>>>> rajkum...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi Manula, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> You can see those values in the same log I attached in >>>>>>>>>>>>>>>>>>>> the previous mail. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam < >>>>>>>>>>>>>>>>>>>> rajkum...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi Chamila, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I couldn't find any large values sent by agent. There >>>>>>>>>>>>>>>>>>>>> are traces for CEP sending large values. Find the cep >>>>>>>>>>>>>>>>>>>>> trace log here [1] >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> 1. >>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika >>>>>>>>>>>>>>>>>>>>> Thantriwatte <manu...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and >>>>>>>>>>>>>>>>>>>>>> see what are the load average values sent from the >>>>>>>>>>>>>>>>>>>>>> agent. If those values >>>>>>>>>>>>>>>>>>>>>> are normal then we can narrow down to the CEP. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thanks ! >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis < >>>>>>>>>>>>>>>>>>>>>> chami...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hi Raj, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time >>>>>>>>>>>>>>>>>>>>>>> period? It will have the events published from the >>>>>>>>>>>>>>>>>>>>>>> agent. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>>>>>> Chamila de Alwis >>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163 >>>>>>>>>>>>>>>>>>>>>>> Blog: code.chamiladealwis.com >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar >>>>>>>>>>>>>>>>>>>>>>> Rajaratnam <rajkum...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent. >>>>>>>>>>>>>>>>>>>>>>>> And faced this very strange scenario, $Subject. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Load average >>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] >>>>>>>>>>>>>>>>>>>>>>>> KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Grad of >>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] >>>>>>>>>>>>>>>>>>>>>>>> -2000.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Load average >>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] >>>>>>>>>>>>>>>>>>>>>>>> KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Grad of >>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Load average >>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] >>>>>>>>>>>>>>>>>>>>>>>> KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Avg >>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] >>>>>>>>>>>>>>>>>>>>>>>> 1.9090909 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Load average >>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] >>>>>>>>>>>>>>>>>>>>>>>> KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Avg >>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> [value] 28.718182 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Second >>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [value] >>>>>>>>>>>>>>>>>>>>>>>> 200000.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Grad of >>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Load average >>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] >>>>>>>>>>>>>>>>>>>>>>>> KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Second >>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> [value] 0.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Load average >>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] >>>>>>>>>>>>>>>>>>>>>>>> KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Second >>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> [value] 0.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Load average >>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] >>>>>>>>>>>>>>>>>>>>>>>> KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Avg >>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] >>>>>>>>>>>>>>>>>>>>>>>> 3.3636363 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Load average >>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] >>>>>>>>>>>>>>>>>>>>>>>> KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Grad of >>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> [value] 0.11670045 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Memory >>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check >>>>>>>>>>>>>>>>>>>>>>>> [kub cluster] KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Avg >>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> [value] 28.7 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Memory >>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check >>>>>>>>>>>>>>>>>>>>>>>> [kub cluster] KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Second >>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [value] 0.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Memory >>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check >>>>>>>>>>>>>>>>>>>>>>>> [kub cluster] KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Avg >>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> [value] 28.7 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Memory >>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check >>>>>>>>>>>>>>>>>>>>>>>> [kub cluster] KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Avg >>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Load average >>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] >>>>>>>>>>>>>>>>>>>>>>>> KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Grad of >>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Load average >>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] >>>>>>>>>>>>>>>>>>>>>>>> KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Avg >>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> [value] 28.7 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Memory >>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check >>>>>>>>>>>>>>>>>>>>>>>> [kub cluster] KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Grad of >>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> [value] 0.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Memory >>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check >>>>>>>>>>>>>>>>>>>>>>>> [kub cluster] KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Second >>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [value] 0.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Memory >>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check >>>>>>>>>>>>>>>>>>>>>>>> [kub cluster] KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Second >>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> [value] 2000000.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Load average >>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] >>>>>>>>>>>>>>>>>>>>>>>> KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Grad of >>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] >>>>>>>>>>>>>>>>>>>>>>>> 13500.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Load average >>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] >>>>>>>>>>>>>>>>>>>>>>>> KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Grad of >>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> [value] 50.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Memory >>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check >>>>>>>>>>>>>>>>>>>>>>>> [kub cluster] KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Avg >>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> [value] 28.738462 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Memory >>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check >>>>>>>>>>>>>>>>>>>>>>>> [kub cluster] KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Avg >>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] >>>>>>>>>>>>>>>>>>>>>>>> 2.3333333 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Load average >>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] >>>>>>>>>>>>>>>>>>>>>>>> KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Grad of >>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Load average >>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] >>>>>>>>>>>>>>>>>>>>>>>> KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Grad of >>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> [value] 0.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Memory >>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check >>>>>>>>>>>>>>>>>>>>>>>> [kub cluster] KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Avg >>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Load average >>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] >>>>>>>>>>>>>>>>>>>>>>>> KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Grad of >>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> [value] 0.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Memory >>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check >>>>>>>>>>>>>>>>>>>>>>>> [kub cluster] KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Avg >>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Load average >>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] >>>>>>>>>>>>>>>>>>>>>>>> KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Avg >>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Load average >>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] >>>>>>>>>>>>>>>>>>>>>>>> KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Grad of >>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] >>>>>>>>>>>>>>>>>>>>>>>> -11000.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Load average >>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] >>>>>>>>>>>>>>>>>>>>>>>> KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Grad of >>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> [value] 0.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Memory >>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check >>>>>>>>>>>>>>>>>>>>>>>> [kub cluster] KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - Avg >>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> [value] 28.8 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} >>>>>>>>>>>>>>>>>>>>>>>> - Memory >>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check >>>>>>>>>>>>>>>>>>>>>>>> [kub cluster] KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} >>>>>>>>>>>>>>>>>>>>>>>> - >>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} >>>>>>>>>>>>>>>>>>>>>>>> - >>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} >>>>>>>>>>>>>>>>>>>>>>>> - >>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - >>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running.. >>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ >>>>>>>>>>>>>>>>>>>>>>>> kubernetesHostClusterId=KubGrp1, >>>>>>>>>>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php] >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - >>>>>>>>>>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> [cluster] : php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> Running minimum rule: >>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [min-check] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [min-check] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> Running obsolete containers >>>>>>>>>>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [obsolete-check] [cluster] >>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} >>>>>>>>>>>>>>>>>>>>>>>> - Minimum >>>>>>>>>>>>>>>>>>>>>>>> check executed for : >>>>>>>>>>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0 >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - >>>>>>>>>>>>>>>>>>>>>>>> flag of rifReset : true flag of >>>>>>>>>>>>>>>>>>>>>>>> memoryConsumptionReset : true flag of >>>>>>>>>>>>>>>>>>>>>>>> loadAverageReset : true >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} >>>>>>>>>>>>>>>>>>>>>>>> - >>>>>>>>>>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 >>>>>>>>>>>>>>>>>>>>>>>> [cluster] : php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> Running scaling rule >>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} >>>>>>>>>>>>>>>>>>>>>>>> - Predicting the >>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second >>>>>>>>>>>>>>>>>>>>>>>> derivative]: 0.0 , [time >>>>>>>>>>>>>>>>>>>>>>>> intervals]: 1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} >>>>>>>>>>>>>>>>>>>>>>>> - Predicting the >>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second >>>>>>>>>>>>>>>>>>>>>>>> derivative]: 0.0 , [time >>>>>>>>>>>>>>>>>>>>>>>> intervals]: 1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} >>>>>>>>>>>>>>>>>>>>>>>> - Predicting the >>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , >>>>>>>>>>>>>>>>>>>>>>>> [second derivative]: >>>>>>>>>>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [scaling] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [scaling] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [scaling] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [scaling] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [scaling] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [scaling] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [scaling] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [scaling] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : >>>>>>>>>>>>>>>>>>>>>>>> 28.799999237060547 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [scaling] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [scaling] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [scaling] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [scaling] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [scaling] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [scaling] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-up action : true >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [scaling] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-down action : false >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [scaling] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [scaling] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas : >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [scaling] Decided to >>>>>>>>>>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - >>>>>>>>>>>>>>>>>>>>>>>> [scaling-up] [cluster] : >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>>>>>>>>>>>>>>>>>>>>>>> - Updating kubernetes replication controller via >>>>>>>>>>>>>>>>>>>>>>>> cloud controller: >>>>>>>>>>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10 >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} >>>>>>>>>>>>>>>>>>>>>>>> - >>>>>>>>>>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for >>>>>>>>>>>>>>>>>>>>>>>> cluster : php.php.domain >>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG >>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} >>>>>>>>>>>>>>>>>>>>>>>> - >>>>>>>>>>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a >>>>>>>>>>>>>>>>>>>>>>>> replication controller >>>>>>>>>>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I couldn't investigate the agent side, since >>>>>>>>>>>>>>>>>>>>>>>> containers are destroyed. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Any idea? >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam >>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc. >>>>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120 >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte >>>>>>>>>>>>>>>>>>>>>> Software Engineer >>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com >>>>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> email : manu...@wso2.com / man...@apache.org >>>>>>>>>>>>>>>>>>>>>> phone : +94 772492511 >>>>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/ >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam >>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc. >>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120 >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam >>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc. >>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte >>>>>>>>>>>>>>>>>>> Software Engineer >>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com >>>>>>>>>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> email : manu...@wso2.com / man...@apache.org >>>>>>>>>>>>>>>>>>> phone : +94 772492511 >>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/ >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte >>>>>>>>>>>>>>>>>> Software Engineer >>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com >>>>>>>>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> email : manu...@wso2.com / man...@apache.org >>>>>>>>>>>>>>>>>> phone : +94 772492511 >>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Rajkumar Rajaratnam >>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc. >>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte >>>>>>>>>>>>>>>> Software Engineer >>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com >>>>>>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> email : manu...@wso2.com / man...@apache.org >>>>>>>>>>>>>>>> phone : +94 772492511 >>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Rajkumar Rajaratnam >>>>>>>>>>>>>>> Software Engineer | WSO2, Inc. >>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120 >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Manula Chathurika Thantriwatte >>>>>>>>>>>>>> Software Engineer >>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com >>>>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>>>> >>>>>>>>>>>>>> email : manu...@wso2.com / man...@apache.org >>>>>>>>>>>>>> phone : +94 772492511 >>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>> Nirmal >>>>>>>>>>>>>> >>>>>>>>>>>>>> Nirmal Fernando. >>>>>>>>>>>>>> PPMC Member & Committer of Apache Stratos, >>>>>>>>>>>>>> Senior Software Engineer, WSO2 Inc. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Imesh Gunaratne >>>>>>>>>>>>> >>>>>>>>>>>>> Technical Lead, WSO2 >>>>>>>>>>>>> Committer & PMC Member, Apache Stratos >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Raj >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> Gayan Gunarathne >>>>>>>>>>> Technical Lead >>>>>>>>>>> WSO2 Inc. (http://wso2.com) >>>>>>>>>>> email : gay...@wso2.com | mobile : +94 766819985 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Best Regards, >>>>>>>>>> Nirmal >>>>>>>>>> >>>>>>>>>> Nirmal Fernando. >>>>>>>>>> PPMC Member & Committer of Apache Stratos, >>>>>>>>>> Senior Software Engineer, WSO2 Inc. >>>>>>>>>> >>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Imesh Gunaratne >>>>>>>>> >>>>>>>>> Technical Lead, WSO2 >>>>>>>>> Committer & PMC Member, Apache Stratos >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Raj >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Raj >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Akila Ravihansa Perera >>>>>> Software Engineer, WSO2 >>>>>> >>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Best Regards, >>>>> Nirmal >>>>> >>>>> Nirmal Fernando. >>>>> PPMC Member & Committer of Apache Stratos, >>>>> Senior Software Engineer, WSO2 Inc. >>>>> >>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>> >>>> >>>> >>>> >>>> -- >>>> Akila Ravihansa Perera >>>> Software Engineer, WSO2 >>>> >>>> Blog: http://ravihansa3000.blogspot.com >>>> >>> >>> >>> >>> -- >>> Best Regards, >>> Nirmal >>> >>> Nirmal Fernando. >>> PPMC Member & Committer of Apache Stratos, >>> Senior Software Engineer, WSO2 Inc. >>> >>> Blog: http://nirmalfdo.blogspot.com/ >>> >> >> >> >> -- >> Imesh Gunaratne >> >> Technical Lead, WSO2 >> Committer & PMC Member, Apache Stratos >> > > > > -- > -- > Lahiru Sandaruwan > Committer and PMC member, Apache Stratos, > Senior Software Engineer, > WSO2 Inc., http://wso2.com > lean.enterprise.middleware > > email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > > -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
