On Wed, Mar 4, 2015 at 11:21 AM, Rajkumar Rajaratnam <rajkum...@wso2.com> wrote:
> As per the offline discussion with Imesh and Lahiru, we have decided to > have three partition selection algorithms. > +1. We can rename current one to 'Weighted RR' Thanks. > > - Round-Robin : The next partition will be selected > > > On Wed, Mar 4, 2015 at 11:14 AM, Gayan Gunarathne <gay...@wso2.com> wrote: > >> Yeah IMO also we need to keep track of the selected index no of the >> partition.I think we may need to persist the selected index to get the RR >> algorithm works correctly. >> >> IMO in the scale up we will increment the selected index up to max index >> no while in scale down it will decrease the selected index until it comes >> to min index no and continue with round robin. >> >> Thanks, >> Gayan >> >> >> >> On Wed, Mar 4, 2015 at 9:42 AM, Rajkumar Rajaratnam <rajkum...@wso2.com> >> wrote: >> >>> Hi Reka, >>> >>> On Wed, Mar 4, 2015 at 1:03 AM, Reka Thirunavukkarasu <r...@wso2.com> >>> wrote: >>> >>>> I think that there is small issue in applying the same RR logic for >>>> scale up and scale down. if it sclaes up, then >>>> >>>> p1 -1, p2-1, p3-1, p1-2, p2-2 >>>> >>>> Then, if we are to continuously scale down, then which partition will >>>> we choose next? Will it be p3? In that way, it won't be equally >>>> distributed..In order to equally distributed, we should choose the >>>> partition in the reverse order in the scale down p2, p1, p3, p2, p1. >>>> >>>> Then as you mentioned, we will need another algorithm specially when >>>> scaling down. We can apply forward RR in scale up and backward RR in >>>> scaling down in order to equally distributed the instances. >>>> >>> >>> Again , if we don't persist current partition index, this logic is also >>> not a pure RR. >>> >>> Current logic is working perfectly according to "equally distribute to >>> partitions" concept in both scale up/down scenarios. But I agree its not a >>> pure RR. >>> >>> IMHO, as Lahiru mentioned, intention of having RR is to equally >>> distribute instances among partitions. So can we stick to this logic for >>> now? If we find any issues, we can improve it later. >>> >>> WDYT? >>> >>>> >>>> Thanks, >>>> Reka >>>> >>>> >>>> >>>> On Tue, Mar 3, 2015 at 12:24 PM, Rajkumar Rajaratnam < >>>> rajkum...@wso2.com> wrote: >>>> >>>>> What I meant in the previous reply is the following scenario. >>>>> >>>>> For eg: >>>>> >>>>> p1, p2 , p3 >>>>> >>>>> If the iteration goes as below: >>>>> selectedIndex = -1 >>>>> >>>>> since all the partitions has 0 instances, choose the first partition >>>>> as the selectedIndex = 1 >>>>> >>>>> p1 - 1 selectedIndex = 1 >>>>> p2 - 1 selectedIndex = 2 >>>>> p3 - 1 selectedIndex = 3 >>>>> p1 - 2 selectedIndex = 1 >>>>> >>>>> scale down happens, instance will be removed from p1 >>>>> p1 = 1, p2 = 1, p3 = 1 >>>>> >>>>> stratos restart >>>>> selectedIndex = -1 >>>>> >>>>> p1 = 2, p2=1, p3=1 choose the first lower instances partition which >>>>> will be p1. >>>>> selectedIndex=1 now, hence not a pure RR. >>>>> >>>>> I guess, it is impossible to implement a pure RR without persisting >>>>> current partition index. >>>>> >>>>> Thanks. >>>>> >>>>> On Wed, Mar 4, 2015 at 12:29 AM, Rajkumar Rajaratnam < >>>>> rajkum...@wso2.com> wrote: >>>>> >>>>>> Hi Reka, >>>>>> >>>>>> There is small issue in the way you suggested above. If you consider >>>>>> scale down scenario, you can't actually say p2 will have min number >>>>>> instances. It can be p1, if scale down happened from p1. So if we restart >>>>>> the stratos, RR will select p1 as next partition right? This is not a >>>>>> pure >>>>>> RR right? >>>>>> >>>>>> Thanks. >>>>>> >>>>>> On Wed, Mar 4, 2015 at 12:24 AM, Reka Thirunavukkarasu <r...@wso2.com >>>>>> > wrote: >>>>>> >>>>>>> Hi Raj, >>>>>>> >>>>>>> I don't think that we will need another algorithm. If we implement >>>>>>> RR in the way that i explained, then it will even make sure to >>>>>>> distribute >>>>>>> the members equally. >>>>>>> >>>>>>> For eg: >>>>>>> >>>>>>> p1, p2 , p3 >>>>>>> >>>>>>> If the iteration goes as below: >>>>>>> selectedIndex = -1 >>>>>>> >>>>>>> since all the partitions has 0 instances, choose the first partition >>>>>>> as the selectedIndex = 1 >>>>>>> >>>>>>> p1 - 1 selectedIndex = 1 >>>>>>> p2 - 1 selectedIndex = 2 >>>>>>> p3 - 1 selectedIndex = 3 >>>>>>> p1 - 2 selectedIndex = 1 >>>>>>> >>>>>>> stratos restart >>>>>>> selectedIndex = -1 >>>>>>> >>>>>>> p1 = 2, p2=1, p3=1 choose the first lower instances partition which >>>>>>> will be p2. (sorry i missed this point earlier) >>>>>>> selectedIndex=2 now >>>>>>> >>>>>>> then iteration can start from where it stops before the restart. >>>>>>> >>>>>>> p2 - 2 selectedIndex = 2 >>>>>>> p3 - 2 selectedIndex = 3 >>>>>>> p1 - 3 selectedIndex = 1 >>>>>>> p2 - 3 selectedIndex = 2 >>>>>>> p3 - 3 selectedIndex = 3 >>>>>>> >>>>>>> If we implement RR in this way, the instances will be equally >>>>>>> distributed as well. >>>>>>> >>>>>>> Thanks, >>>>>>> Reka >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Reka >>>>>>> >>>>>>> On Tue, Mar 3, 2015 at 11:43 AM, Rajkumar Rajaratnam < >>>>>>> rajkum...@wso2.com> wrote: >>>>>>> >>>>>>>> So I guess we have three algorithms :) >>>>>>>> >>>>>>>> 1. one-after-another (It will go to next partition only if the >>>>>>>> current partition is full) >>>>>>>> 2. round-robin (if we implement as Reka explained above. It will >>>>>>>> always select the next partition) >>>>>>>> 3. name-should-be-decided (this will work as Lahiru explained >>>>>>>> above. It will always distribute the members in the partitions equally) >>>>>>>> >>>>>>>> WDYT? >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> On Wed, Mar 4, 2015 at 12:01 AM, Reka Thirunavukkarasu < >>>>>>>> r...@wso2.com> wrote: >>>>>>>> >>>>>>>>> I think that if the selectedIndex is initial value as -1, then RR >>>>>>>>> should traverse through all the partitions and find out who has the >>>>>>>>> minimum >>>>>>>>> instances and choose that partition as the selectedIndex in the first >>>>>>>>> iteration(may be we can exc). Then from the second iteration onwards, >>>>>>>>> it >>>>>>>>> can increase the selectedIndex by 1 to choose the next adjacent >>>>>>>>> partition. >>>>>>>>> This will help in the stratos restart as we don't persist the >>>>>>>>> selectedIndex. WDYT? >>>>>>>>> >>>>>>>>> On Tue, Mar 3, 2015 at 10:53 AM, Lahiru Sandaruwan < >>>>>>>>> lahi...@wso2.com> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Mar 3, 2015 at 11:15 PM, Rajkumar Rajaratnam < >>>>>>>>>> rajkum...@wso2.com> wrote: >>>>>>>>>> >>>>>>>>>>> Problem is lowestInstanceCount is initially taking 0. Instead it >>>>>>>>>>> should take the non terminated member count of 1st partition. >>>>>>>>>>> >>>>>>>>>>> To Fix, >>>>>>>>>>> >>>>>>>>>>> int selectedIndex = 0; >>>>>>>>>>> int lowestInstanceCount = >>>>>>>>>>> partitionContexts[0].getNonTerminatedMemberCount(); >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> +1. I was thinking the same. We can start with first partition >>>>>>>>>> count. >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>>> wdyt? >>>>>>>>>>> >>>>>>>>>>> Thanks. >>>>>>>>>>> >>>>>>>>>>> On Tue, Mar 3, 2015 at 10:57 PM, Rajkumar Rajaratnam < >>>>>>>>>>> rajkum...@wso2.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Lahiru, >>>>>>>>>>>> >>>>>>>>>>>> Actually, current logic is not working as your example. This >>>>>>>>>>>> logic only uses the first partition. This will never use any other >>>>>>>>>>>> partitions. See my comments within the following code. >>>>>>>>>>>> >>>>>>>>>>>> public PartitionContext >>>>>>>>>>>> getNextScaleUpPartitionContext(PartitionContext[] >>>>>>>>>>>> partitionContexts) { >>>>>>>>>>>> >>>>>>>>>>>> int selectedIndex = 0; >>>>>>>>>>>> int lowestInstanceCount = 0; >>>>>>>>>>>> >>>>>>>>>>>> for(int partitionIndex = 0; partitionIndex < >>>>>>>>>>>> partitionContexts.length - 1; partitionIndex++) { >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> if(partitionContexts[partitionIndex].getNonTerminatedMemberCount() >>>>>>>>>>>> < >>>>>>>>>>>> lowestInstanceCount) { >>>>>>>>>>>> >>>>>>>>>>>> *// this if condition is always false, because non terminated >>>>>>>>>>>> count is always >=0* >>>>>>>>>>>> >>>>>>>>>>>> lowestInstanceCount = >>>>>>>>>>>> partitionContexts[partitionIndex].getNonTerminatedMemberCount(); >>>>>>>>>>>> selectedIndex = partitionIndex; >>>>>>>>>>>> } >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> *// selected index is 0 always, so round-robin will always use >>>>>>>>>>>> first partition* >>>>>>>>>>>> >>>>>>>>>>>> if(partitionContexts[selectedIndex].getNonTerminatedMemberCount() < >>>>>>>>>>>> partitionContexts[selectedIndex].getMax()) { >>>>>>>>>>>> >>>>>>>>>>>> if(log.isDebugEnabled()){ >>>>>>>>>>>> log.debug(String.format("[round-robin >>>>>>>>>>>> algorithm] [scale-up] [partition] %s has space to create members. >>>>>>>>>>>> " + >>>>>>>>>>>> "[non terminated count] %s [max] %s" >>>>>>>>>>>> , >>>>>>>>>>>> partitionContexts[selectedIndex].getPartitionId(), >>>>>>>>>>>> >>>>>>>>>>>> partitionContexts[selectedIndex].getNonTerminatedMemberCount(), >>>>>>>>>>>> >>>>>>>>>>>> partitionContexts[selectedIndex].getMax())); >>>>>>>>>>>> } >>>>>>>>>>>> return partitionContexts[selectedIndex]; >>>>>>>>>>>> } else { >>>>>>>>>>>> >>>>>>>>>>>> return null; >>>>>>>>>>>> } >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> May be I am missing something here. >>>>>>>>>>>> >>>>>>>>>>>> Thanks. >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Mar 3, 2015 at 10:42 PM, Lahiru Sandaruwan < >>>>>>>>>>>> lahi...@wso2.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Raj, >>>>>>>>>>>>> >>>>>>>>>>>>> Earlier in 4.0.0 release, we have been using the partition >>>>>>>>>>>>> index. If that to be worked correctly we should persist the index >>>>>>>>>>>>> for each >>>>>>>>>>>>> cluster. >>>>>>>>>>>>> >>>>>>>>>>>>> IMO there is a better way to execute the round-robin method as >>>>>>>>>>>>> follows, >>>>>>>>>>>>> >>>>>>>>>>>>> The intention of round robin algorithm is to distribute the >>>>>>>>>>>>> members in the partitions equally. Current method works as per the >>>>>>>>>>>>> following example. >>>>>>>>>>>>> >>>>>>>>>>>>> Say we have 3 partitions, p1, p2, and p3 and max of each >>>>>>>>>>>>> partition is 2. >>>>>>>>>>>>> >>>>>>>>>>>>> So according to the algorithm, it will select the first >>>>>>>>>>>>> partition who has the lowest member count. >>>>>>>>>>>>> >>>>>>>>>>>>> Iteration1: It will select p1. >>>>>>>>>>>>> Iteration2: It will select p2. >>>>>>>>>>>>> Iteration3: It will select p3. >>>>>>>>>>>>> Iteration4: It will select p1. >>>>>>>>>>>>> Iteration5: It will select p2. >>>>>>>>>>>>> .... >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Mar 3, 2015 at 10:22 PM, Rajkumar Rajaratnam < >>>>>>>>>>>>> rajkum...@wso2.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Devs, >>>>>>>>>>>>>> >>>>>>>>>>>>>> It seems to me that there is a bug in round-robin >>>>>>>>>>>>>> implementation of partition algorithm. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://github.com/apache/stratos/blob/0b7734f4c9f1444d064fec93bf9ac59a5883faf2/components/org.apache.stratos.autoscaler/src/main/java/org/apache/stratos/autoscaler/algorithm/RoundRobin.java#L43-L64 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Shouldn't it work based on current partition index? For >>>>>>>>>>>>>> example, if the most recent instance was spin up in partition-1, >>>>>>>>>>>>>> the next >>>>>>>>>>>>>> instance should be spin up in partition-2 and so on. According >>>>>>>>>>>>>> to current >>>>>>>>>>>>>> logic what is happening is, all the instance are spin up in >>>>>>>>>>>>>> partition-1. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Please correct me If I am wrong. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Rajkumar Rajaratnam >>>>>>>>>>>>>> Committer & PMC Member, Apache Stratos >>>>>>>>>>>>>> Software Engineer, WSO2 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Mobile : +94777568639 >>>>>>>>>>>>>> Blog : rajkumarr.com >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> -- >>>>>>>>>>>>> Lahiru Sandaruwan >>>>>>>>>>>>> Committer and PMC member, Apache Stratos, >>>>>>>>>>>>> Senior Software Engineer, >>>>>>>>>>>>> WSO2 Inc., http://wso2.com >>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>> >>>>>>>>>>>>> email: lahi...@wso2.com blog: >>>>>>>>>>>>> http://lahiruwrites.blogspot.com/ >>>>>>>>>>>>> linked-in: >>>>>>>>>>>>> http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Rajkumar Rajaratnam >>>>>>>>>>>> Committer & PMC Member, Apache Stratos >>>>>>>>>>>> Software Engineer, WSO2 >>>>>>>>>>>> >>>>>>>>>>>> Mobile : +94777568639 >>>>>>>>>>>> Blog : rajkumarr.com >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Rajkumar Rajaratnam >>>>>>>>>>> Committer & PMC Member, Apache Stratos >>>>>>>>>>> Software Engineer, WSO2 >>>>>>>>>>> >>>>>>>>>>> Mobile : +94777568639 >>>>>>>>>>> Blog : rajkumarr.com >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> -- >>>>>>>>>> Lahiru Sandaruwan >>>>>>>>>> Committer and PMC member, Apache Stratos, >>>>>>>>>> Senior Software Engineer, >>>>>>>>>> WSO2 Inc., http://wso2.com >>>>>>>>>> lean.enterprise.middleware >>>>>>>>>> >>>>>>>>>> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ >>>>>>>>>> linked-in: >>>>>>>>>> http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Reka Thirunavukkarasu >>>>>>>>> Senior Software Engineer, >>>>>>>>> WSO2, Inc.:http://wso2.com, >>>>>>>>> Mobile: +94776442007 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Rajkumar Rajaratnam >>>>>>>> Committer & PMC Member, Apache Stratos >>>>>>>> Software Engineer, WSO2 >>>>>>>> >>>>>>>> Mobile : +94777568639 >>>>>>>> Blog : rajkumarr.com >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Reka Thirunavukkarasu >>>>>>> Senior Software Engineer, >>>>>>> WSO2, Inc.:http://wso2.com, >>>>>>> Mobile: +94776442007 >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Rajkumar Rajaratnam >>>>>> Committer & PMC Member, Apache Stratos >>>>>> Software Engineer, WSO2 >>>>>> >>>>>> Mobile : +94777568639 >>>>>> Blog : rajkumarr.com >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Rajkumar Rajaratnam >>>>> Committer & PMC Member, Apache Stratos >>>>> Software Engineer, WSO2 >>>>> >>>>> Mobile : +94777568639 >>>>> Blog : rajkumarr.com >>>>> >>>> >>>> >>>> >>>> -- >>>> Reka Thirunavukkarasu >>>> Senior Software Engineer, >>>> WSO2, Inc.:http://wso2.com, >>>> Mobile: +94776442007 >>>> >>>> >>>> >>> >>> >>> -- >>> Rajkumar Rajaratnam >>> Committer & PMC Member, Apache Stratos >>> Software Engineer, WSO2 >>> >>> Mobile : +94777568639 >>> Blog : rajkumarr.com >>> >> >> >> >> -- >> >> Gayan Gunarathne >> Technical Lead >> WSO2 Inc. (http://wso2.com) >> email : gay...@wso2.com | mobile : +94 766819985 >> >> > > > > -- > Rajkumar Rajaratnam > Committer & PMC Member, Apache Stratos > Software Engineer, WSO2 > > Mobile : +94777568639 > Blog : rajkumarr.com > -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146