Hi guys, Interesting discussion. I think we can live with 2 algorithms. Because user might be confused if we introduced a variations with small changes to RR and user might not need to fine tune the algorithms that much.
On Wed, Mar 4, 2015 at 1:03 AM, Reka Thirunavukkarasu <r...@wso2.com> wrote: > I think that there is small issue in applying the same RR logic for scale > up and scale down. if it sclaes up, then > > p1 -1, p2-1, p3-1, p1-2, p2-2 > > Then, if we are to continuously scale down, then which partition will we > choose next? Will it be p3? In that way, it won't be equally > distributed..In order to equally distributed, we should choose the > partition in the reverse order in the scale down p2, p1, p3, p2, p1. > Yes, the reverse order should be exactly this IMO. > > Then as you mentioned, we will need another algorithm specially when > scaling down. We can apply forward RR in scale up and backward RR in > scaling down in order to equally distributed the instances. > > If we are going with equally distributed concept, we are already doing this in scale down section. Wdyt? Thanks. > Thanks, > Reka > > > > On Tue, Mar 3, 2015 at 12:24 PM, Rajkumar Rajaratnam <rajkum...@wso2.com> > wrote: > >> What I meant in the previous reply is the following scenario. >> >> For eg: >> >> p1, p2 , p3 >> >> If the iteration goes as below: >> selectedIndex = -1 >> >> since all the partitions has 0 instances, choose the first partition as >> the selectedIndex = 1 >> >> p1 - 1 selectedIndex = 1 >> p2 - 1 selectedIndex = 2 >> p3 - 1 selectedIndex = 3 >> p1 - 2 selectedIndex = 1 >> >> scale down happens, instance will be removed from p1 >> p1 = 1, p2 = 1, p3 = 1 >> >> stratos restart >> selectedIndex = -1 >> >> p1 = 2, p2=1, p3=1 choose the first lower instances partition which will >> be p1. >> selectedIndex=1 now, hence not a pure RR. >> >> I guess, it is impossible to implement a pure RR without persisting >> current partition index. >> >> Thanks. >> >> On Wed, Mar 4, 2015 at 12:29 AM, Rajkumar Rajaratnam <rajkum...@wso2.com> >> wrote: >> >>> Hi Reka, >>> >>> There is small issue in the way you suggested above. If you consider >>> scale down scenario, you can't actually say p2 will have min number >>> instances. It can be p1, if scale down happened from p1. So if we restart >>> the stratos, RR will select p1 as next partition right? This is not a pure >>> RR right? >>> >>> Thanks. >>> >>> On Wed, Mar 4, 2015 at 12:24 AM, Reka Thirunavukkarasu <r...@wso2.com> >>> wrote: >>> >>>> Hi Raj, >>>> >>>> I don't think that we will need another algorithm. If we implement RR >>>> in the way that i explained, then it will even make sure to distribute the >>>> members equally. >>>> >>>> For eg: >>>> >>>> p1, p2 , p3 >>>> >>>> If the iteration goes as below: >>>> selectedIndex = -1 >>>> >>>> since all the partitions has 0 instances, choose the first partition as >>>> the selectedIndex = 1 >>>> >>>> p1 - 1 selectedIndex = 1 >>>> p2 - 1 selectedIndex = 2 >>>> p3 - 1 selectedIndex = 3 >>>> p1 - 2 selectedIndex = 1 >>>> >>>> stratos restart >>>> selectedIndex = -1 >>>> >>>> p1 = 2, p2=1, p3=1 choose the first lower instances partition which >>>> will be p2. (sorry i missed this point earlier) >>>> selectedIndex=2 now >>>> >>>> then iteration can start from where it stops before the restart. >>>> >>>> p2 - 2 selectedIndex = 2 >>>> p3 - 2 selectedIndex = 3 >>>> p1 - 3 selectedIndex = 1 >>>> p2 - 3 selectedIndex = 2 >>>> p3 - 3 selectedIndex = 3 >>>> >>>> If we implement RR in this way, the instances will be equally >>>> distributed as well. >>>> >>>> Thanks, >>>> Reka >>>> >>>> >>>> Thanks, >>>> Reka >>>> >>>> On Tue, Mar 3, 2015 at 11:43 AM, Rajkumar Rajaratnam < >>>> rajkum...@wso2.com> wrote: >>>> >>>>> So I guess we have three algorithms :) >>>>> >>>>> 1. one-after-another (It will go to next partition only if the current >>>>> partition is full) >>>>> 2. round-robin (if we implement as Reka explained above. It will >>>>> always select the next partition) >>>>> 3. name-should-be-decided (this will work as Lahiru explained above. >>>>> It will always distribute the members in the partitions equally) >>>>> >>>>> WDYT? >>>>> >>>>> Thanks. >>>>> >>>>> On Wed, Mar 4, 2015 at 12:01 AM, Reka Thirunavukkarasu <r...@wso2.com> >>>>> wrote: >>>>> >>>>>> I think that if the selectedIndex is initial value as -1, then RR >>>>>> should traverse through all the partitions and find out who has the >>>>>> minimum >>>>>> instances and choose that partition as the selectedIndex in the first >>>>>> iteration(may be we can exc). Then from the second iteration onwards, it >>>>>> can increase the selectedIndex by 1 to choose the next adjacent >>>>>> partition. >>>>>> This will help in the stratos restart as we don't persist the >>>>>> selectedIndex. WDYT? >>>>>> >>>>>> On Tue, Mar 3, 2015 at 10:53 AM, Lahiru Sandaruwan <lahi...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Mar 3, 2015 at 11:15 PM, Rajkumar Rajaratnam < >>>>>>> rajkum...@wso2.com> wrote: >>>>>>> >>>>>>>> Problem is lowestInstanceCount is initially taking 0. Instead it >>>>>>>> should take the non terminated member count of 1st partition. >>>>>>>> >>>>>>>> To Fix, >>>>>>>> >>>>>>>> int selectedIndex = 0; >>>>>>>> int lowestInstanceCount = >>>>>>>> partitionContexts[0].getNonTerminatedMemberCount(); >>>>>>>> >>>>>>>> >>>>>>> +1. I was thinking the same. We can start with first partition count. >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>>> wdyt? >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> On Tue, Mar 3, 2015 at 10:57 PM, Rajkumar Rajaratnam < >>>>>>>> rajkum...@wso2.com> wrote: >>>>>>>> >>>>>>>>> Hi Lahiru, >>>>>>>>> >>>>>>>>> Actually, current logic is not working as your example. This logic >>>>>>>>> only uses the first partition. This will never use any other >>>>>>>>> partitions. >>>>>>>>> See my comments within the following code. >>>>>>>>> >>>>>>>>> public PartitionContext >>>>>>>>> getNextScaleUpPartitionContext(PartitionContext[] partitionContexts) { >>>>>>>>> >>>>>>>>> int selectedIndex = 0; >>>>>>>>> int lowestInstanceCount = 0; >>>>>>>>> >>>>>>>>> for(int partitionIndex = 0; partitionIndex < >>>>>>>>> partitionContexts.length - 1; partitionIndex++) { >>>>>>>>> >>>>>>>>> >>>>>>>>> if(partitionContexts[partitionIndex].getNonTerminatedMemberCount() < >>>>>>>>> lowestInstanceCount) { >>>>>>>>> >>>>>>>>> *// this if condition is always false, because non terminated >>>>>>>>> count is always >=0* >>>>>>>>> >>>>>>>>> lowestInstanceCount = >>>>>>>>> partitionContexts[partitionIndex].getNonTerminatedMemberCount(); >>>>>>>>> selectedIndex = partitionIndex; >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> *// selected index is 0 always, so round-robin will always use >>>>>>>>> first partition* >>>>>>>>> >>>>>>>>> if(partitionContexts[selectedIndex].getNonTerminatedMemberCount() < >>>>>>>>> partitionContexts[selectedIndex].getMax()) { >>>>>>>>> >>>>>>>>> if(log.isDebugEnabled()){ >>>>>>>>> log.debug(String.format("[round-robin algorithm] >>>>>>>>> [scale-up] [partition] %s has space to create members. " + >>>>>>>>> "[non terminated count] %s [max] %s" >>>>>>>>> , >>>>>>>>> partitionContexts[selectedIndex].getPartitionId(), >>>>>>>>> >>>>>>>>> partitionContexts[selectedIndex].getNonTerminatedMemberCount(), >>>>>>>>> >>>>>>>>> partitionContexts[selectedIndex].getMax())); >>>>>>>>> } >>>>>>>>> return partitionContexts[selectedIndex]; >>>>>>>>> } else { >>>>>>>>> >>>>>>>>> return null; >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> May be I am missing something here. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> On Tue, Mar 3, 2015 at 10:42 PM, Lahiru Sandaruwan < >>>>>>>>> lahi...@wso2.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Raj, >>>>>>>>>> >>>>>>>>>> Earlier in 4.0.0 release, we have been using the partition index. >>>>>>>>>> If that to be worked correctly we should persist the index for each >>>>>>>>>> cluster. >>>>>>>>>> >>>>>>>>>> IMO there is a better way to execute the round-robin method as >>>>>>>>>> follows, >>>>>>>>>> >>>>>>>>>> The intention of round robin algorithm is to distribute the >>>>>>>>>> members in the partitions equally. Current method works as per the >>>>>>>>>> following example. >>>>>>>>>> >>>>>>>>>> Say we have 3 partitions, p1, p2, and p3 and max of each >>>>>>>>>> partition is 2. >>>>>>>>>> >>>>>>>>>> So according to the algorithm, it will select the first partition >>>>>>>>>> who has the lowest member count. >>>>>>>>>> >>>>>>>>>> Iteration1: It will select p1. >>>>>>>>>> Iteration2: It will select p2. >>>>>>>>>> Iteration3: It will select p3. >>>>>>>>>> Iteration4: It will select p1. >>>>>>>>>> Iteration5: It will select p2. >>>>>>>>>> .... >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Mar 3, 2015 at 10:22 PM, Rajkumar Rajaratnam < >>>>>>>>>> rajkum...@wso2.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Devs, >>>>>>>>>>> >>>>>>>>>>> It seems to me that there is a bug in round-robin implementation >>>>>>>>>>> of partition algorithm. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> https://github.com/apache/stratos/blob/0b7734f4c9f1444d064fec93bf9ac59a5883faf2/components/org.apache.stratos.autoscaler/src/main/java/org/apache/stratos/autoscaler/algorithm/RoundRobin.java#L43-L64 >>>>>>>>>>> >>>>>>>>>>> Shouldn't it work based on current partition index? For example, >>>>>>>>>>> if the most recent instance was spin up in partition-1, the next >>>>>>>>>>> instance >>>>>>>>>>> should be spin up in partition-2 and so on. According to current >>>>>>>>>>> logic what >>>>>>>>>>> is happening is, all the instance are spin up in partition-1. >>>>>>>>>>> >>>>>>>>>>> Please correct me If I am wrong. >>>>>>>>>>> >>>>>>>>>>> Thanks. >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Rajkumar Rajaratnam >>>>>>>>>>> Committer & PMC Member, Apache Stratos >>>>>>>>>>> Software Engineer, WSO2 >>>>>>>>>>> >>>>>>>>>>> Mobile : +94777568639 >>>>>>>>>>> Blog : rajkumarr.com >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> -- >>>>>>>>>> Lahiru Sandaruwan >>>>>>>>>> Committer and PMC member, Apache Stratos, >>>>>>>>>> Senior Software Engineer, >>>>>>>>>> WSO2 Inc., http://wso2.com >>>>>>>>>> lean.enterprise.middleware >>>>>>>>>> >>>>>>>>>> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ >>>>>>>>>> linked-in: >>>>>>>>>> http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Rajkumar Rajaratnam >>>>>>>>> Committer & PMC Member, Apache Stratos >>>>>>>>> Software Engineer, WSO2 >>>>>>>>> >>>>>>>>> Mobile : +94777568639 >>>>>>>>> Blog : rajkumarr.com >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Rajkumar Rajaratnam >>>>>>>> Committer & PMC Member, Apache Stratos >>>>>>>> Software Engineer, WSO2 >>>>>>>> >>>>>>>> Mobile : +94777568639 >>>>>>>> Blog : rajkumarr.com >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> -- >>>>>>> Lahiru Sandaruwan >>>>>>> Committer and PMC member, Apache Stratos, >>>>>>> Senior Software Engineer, >>>>>>> WSO2 Inc., http://wso2.com >>>>>>> lean.enterprise.middleware >>>>>>> >>>>>>> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ >>>>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Reka Thirunavukkarasu >>>>>> Senior Software Engineer, >>>>>> WSO2, Inc.:http://wso2.com, >>>>>> Mobile: +94776442007 >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Rajkumar Rajaratnam >>>>> Committer & PMC Member, Apache Stratos >>>>> Software Engineer, WSO2 >>>>> >>>>> Mobile : +94777568639 >>>>> Blog : rajkumarr.com >>>>> >>>> >>>> >>>> >>>> -- >>>> Reka Thirunavukkarasu >>>> Senior Software Engineer, >>>> WSO2, Inc.:http://wso2.com, >>>> Mobile: +94776442007 >>>> >>>> >>>> >>> >>> >>> -- >>> Rajkumar Rajaratnam >>> Committer & PMC Member, Apache Stratos >>> Software Engineer, WSO2 >>> >>> Mobile : +94777568639 >>> Blog : rajkumarr.com >>> >> >> >> >> -- >> Rajkumar Rajaratnam >> Committer & PMC Member, Apache Stratos >> Software Engineer, WSO2 >> >> Mobile : +94777568639 >> Blog : rajkumarr.com >> > > > > -- > Reka Thirunavukkarasu > Senior Software Engineer, > WSO2, Inc.:http://wso2.com, > Mobile: +94776442007 > > > -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146