Hi guys,
Interesting discussion. I think we can live with 2 algorithms. Because user
might be confused if we introduced a variations with small changes to RR
and user might not need to fine tune the algorithms that much.
On Wed, Mar 4, 2015 at 1:03 AM, Reka Thirunavukkarasu <r...@wso2.com> wrote:
> I think that there is small issue in applying the same RR logic for scale
> up and scale down. if it sclaes up, then
>
> p1 -1, p2-1, p3-1, p1-2, p2-2
>
> Then, if we are to continuously scale down, then which partition will we
> choose next? Will it be p3? In that way, it won't be equally
> distributed..In order to equally distributed, we should choose the
> partition in the reverse order in the scale down p2, p1, p3, p2, p1.
>
Yes, the reverse order should be exactly this IMO.
>
> Then as you mentioned, we will need another algorithm specially when
> scaling down. We can apply forward RR in scale up and backward RR in
> scaling down in order to equally distributed the instances.
>
>
If we are going with equally distributed concept, we are already doing this
in scale down section.
Wdyt?
Thanks.
> Thanks,
> Reka
>
>
>
> On Tue, Mar 3, 2015 at 12:24 PM, Rajkumar Rajaratnam <rajkum...@wso2.com>
> wrote:
>
>> What I meant in the previous reply is the following scenario.
>>
>> For eg:
>>
>> p1, p2 , p3
>>
>> If the iteration goes as below:
>> selectedIndex = -1
>>
>> since all the partitions has 0 instances, choose the first partition as
>> the selectedIndex = 1
>>
>> p1 - 1 selectedIndex = 1
>> p2 - 1 selectedIndex = 2
>> p3 - 1 selectedIndex = 3
>> p1 - 2 selectedIndex = 1
>>
>> scale down happens, instance will be removed from p1
>> p1 = 1, p2 = 1, p3 = 1
>>
>> stratos restart
>> selectedIndex = -1
>>
>> p1 = 2, p2=1, p3=1 choose the first lower instances partition which will
>> be p1.
>> selectedIndex=1 now, hence not a pure RR.
>>
>> I guess, it is impossible to implement a pure RR without persisting
>> current partition index.
>>
>> Thanks.
>>
>> On Wed, Mar 4, 2015 at 12:29 AM, Rajkumar Rajaratnam <rajkum...@wso2.com>
>> wrote:
>>
>>> Hi Reka,
>>>
>>> There is small issue in the way you suggested above. If you consider
>>> scale down scenario, you can't actually say p2 will have min number
>>> instances. It can be p1, if scale down happened from p1. So if we restart
>>> the stratos, RR will select p1 as next partition right? This is not a pure
>>> RR right?
>>>
>>> Thanks.
>>>
>>> On Wed, Mar 4, 2015 at 12:24 AM, Reka Thirunavukkarasu <r...@wso2.com>
>>> wrote:
>>>
>>>> Hi Raj,
>>>>
>>>> I don't think that we will need another algorithm. If we implement RR
>>>> in the way that i explained, then it will even make sure to distribute the
>>>> members equally.
>>>>
>>>> For eg:
>>>>
>>>> p1, p2 , p3
>>>>
>>>> If the iteration goes as below:
>>>> selectedIndex = -1
>>>>
>>>> since all the partitions has 0 instances, choose the first partition as
>>>> the selectedIndex = 1
>>>>
>>>> p1 - 1 selectedIndex = 1
>>>> p2 - 1 selectedIndex = 2
>>>> p3 - 1 selectedIndex = 3
>>>> p1 - 2 selectedIndex = 1
>>>>
>>>> stratos restart
>>>> selectedIndex = -1
>>>>
>>>> p1 = 2, p2=1, p3=1 choose the first lower instances partition which
>>>> will be p2. (sorry i missed this point earlier)
>>>> selectedIndex=2 now
>>>>
>>>> then iteration can start from where it stops before the restart.
>>>>
>>>> p2 - 2 selectedIndex = 2
>>>> p3 - 2 selectedIndex = 3
>>>> p1 - 3 selectedIndex = 1
>>>> p2 - 3 selectedIndex = 2
>>>> p3 - 3 selectedIndex = 3
>>>>
>>>> If we implement RR in this way, the instances will be equally
>>>> distributed as well.
>>>>
>>>> Thanks,
>>>> Reka
>>>>
>>>>
>>>> Thanks,
>>>> Reka
>>>>
>>>> On Tue, Mar 3, 2015 at 11:43 AM, Rajkumar Rajaratnam <
>>>> rajkum...@wso2.com> wrote:
>>>>
>>>>> So I guess we have three algorithms :)
>>>>>
>>>>> 1. one-after-another (It will go to next partition only if the current
>>>>> partition is full)
>>>>> 2. round-robin (if we implement as Reka explained above. It will
>>>>> always select the next partition)
>>>>> 3. name-should-be-decided (this will work as Lahiru explained above.
>>>>> It will always distribute the members in the partitions equally)
>>>>>
>>>>> WDYT?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> On Wed, Mar 4, 2015 at 12:01 AM, Reka Thirunavukkarasu <r...@wso2.com>
>>>>> wrote:
>>>>>
>>>>>> I think that if the selectedIndex is initial value as -1, then RR
>>>>>> should traverse through all the partitions and find out who has the
>>>>>> minimum
>>>>>> instances and choose that partition as the selectedIndex in the first
>>>>>> iteration(may be we can exc). Then from the second iteration onwards, it
>>>>>> can increase the selectedIndex by 1 to choose the next adjacent
>>>>>> partition.
>>>>>> This will help in the stratos restart as we don't persist the
>>>>>> selectedIndex. WDYT?
>>>>>>
>>>>>> On Tue, Mar 3, 2015 at 10:53 AM, Lahiru Sandaruwan <lahi...@wso2.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Mar 3, 2015 at 11:15 PM, Rajkumar Rajaratnam <
>>>>>>> rajkum...@wso2.com> wrote:
>>>>>>>
>>>>>>>> Problem is lowestInstanceCount is initially taking 0. Instead it
>>>>>>>> should take the non terminated member count of 1st partition.
>>>>>>>>
>>>>>>>> To Fix,
>>>>>>>>
>>>>>>>> int selectedIndex = 0;
>>>>>>>> int lowestInstanceCount =
>>>>>>>> partitionContexts[0].getNonTerminatedMemberCount();
>>>>>>>>
>>>>>>>>
>>>>>>> +1. I was thinking the same. We can start with first partition count.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>>> wdyt?
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> On Tue, Mar 3, 2015 at 10:57 PM, Rajkumar Rajaratnam <
>>>>>>>> rajkum...@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Lahiru,
>>>>>>>>>
>>>>>>>>> Actually, current logic is not working as your example. This logic
>>>>>>>>> only uses the first partition. This will never use any other
>>>>>>>>> partitions.
>>>>>>>>> See my comments within the following code.
>>>>>>>>>
>>>>>>>>> public PartitionContext
>>>>>>>>> getNextScaleUpPartitionContext(PartitionContext[] partitionContexts) {
>>>>>>>>>
>>>>>>>>> int selectedIndex = 0;
>>>>>>>>> int lowestInstanceCount = 0;
>>>>>>>>>
>>>>>>>>> for(int partitionIndex = 0; partitionIndex <
>>>>>>>>> partitionContexts.length - 1; partitionIndex++) {
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> if(partitionContexts[partitionIndex].getNonTerminatedMemberCount() <
>>>>>>>>> lowestInstanceCount) {
>>>>>>>>>
>>>>>>>>> *// this if condition is always false, because non terminated
>>>>>>>>> count is always >=0*
>>>>>>>>>
>>>>>>>>> lowestInstanceCount =
>>>>>>>>> partitionContexts[partitionIndex].getNonTerminatedMemberCount();
>>>>>>>>> selectedIndex = partitionIndex;
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> *// selected index is 0 always, so round-robin will always use
>>>>>>>>> first partition*
>>>>>>>>>
>>>>>>>>> if(partitionContexts[selectedIndex].getNonTerminatedMemberCount() <
>>>>>>>>> partitionContexts[selectedIndex].getMax()) {
>>>>>>>>>
>>>>>>>>> if(log.isDebugEnabled()){
>>>>>>>>> log.debug(String.format("[round-robin algorithm]
>>>>>>>>> [scale-up] [partition] %s has space to create members. " +
>>>>>>>>> "[non terminated count] %s [max] %s"
>>>>>>>>> ,
>>>>>>>>> partitionContexts[selectedIndex].getPartitionId(),
>>>>>>>>>
>>>>>>>>> partitionContexts[selectedIndex].getNonTerminatedMemberCount(),
>>>>>>>>>
>>>>>>>>> partitionContexts[selectedIndex].getMax()));
>>>>>>>>> }
>>>>>>>>> return partitionContexts[selectedIndex];
>>>>>>>>> } else {
>>>>>>>>>
>>>>>>>>> return null;
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> May be I am missing something here.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> On Tue, Mar 3, 2015 at 10:42 PM, Lahiru Sandaruwan <
>>>>>>>>> lahi...@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Raj,
>>>>>>>>>>
>>>>>>>>>> Earlier in 4.0.0 release, we have been using the partition index.
>>>>>>>>>> If that to be worked correctly we should persist the index for each
>>>>>>>>>> cluster.
>>>>>>>>>>
>>>>>>>>>> IMO there is a better way to execute the round-robin method as
>>>>>>>>>> follows,
>>>>>>>>>>
>>>>>>>>>> The intention of round robin algorithm is to distribute the
>>>>>>>>>> members in the partitions equally. Current method works as per the
>>>>>>>>>> following example.
>>>>>>>>>>
>>>>>>>>>> Say we have 3 partitions, p1, p2, and p3 and max of each
>>>>>>>>>> partition is 2.
>>>>>>>>>>
>>>>>>>>>> So according to the algorithm, it will select the first partition
>>>>>>>>>> who has the lowest member count.
>>>>>>>>>>
>>>>>>>>>> Iteration1: It will select p1.
>>>>>>>>>> Iteration2: It will select p2.
>>>>>>>>>> Iteration3: It will select p3.
>>>>>>>>>> Iteration4: It will select p1.
>>>>>>>>>> Iteration5: It will select p2.
>>>>>>>>>> ....
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Mar 3, 2015 at 10:22 PM, Rajkumar Rajaratnam <
>>>>>>>>>> rajkum...@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Devs,
>>>>>>>>>>>
>>>>>>>>>>> It seems to me that there is a bug in round-robin implementation
>>>>>>>>>>> of partition algorithm.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/apache/stratos/blob/0b7734f4c9f1444d064fec93bf9ac59a5883faf2/components/org.apache.stratos.autoscaler/src/main/java/org/apache/stratos/autoscaler/algorithm/RoundRobin.java#L43-L64
>>>>>>>>>>>
>>>>>>>>>>> Shouldn't it work based on current partition index? For example,
>>>>>>>>>>> if the most recent instance was spin up in partition-1, the next
>>>>>>>>>>> instance
>>>>>>>>>>> should be spin up in partition-2 and so on. According to current
>>>>>>>>>>> logic what
>>>>>>>>>>> is happening is, all the instance are spin up in partition-1.
>>>>>>>>>>>
>>>>>>>>>>> Please correct me If I am wrong.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>> Software Engineer, WSO2
>>>>>>>>>>>
>>>>>>>>>>> Mobile : +94777568639
>>>>>>>>>>> Blog : rajkumarr.com
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> --
>>>>>>>>>> Lahiru Sandaruwan
>>>>>>>>>> Committer and PMC member, Apache Stratos,
>>>>>>>>>> Senior Software Engineer,
>>>>>>>>>> WSO2 Inc., http://wso2.com
>>>>>>>>>> lean.enterprise.middleware
>>>>>>>>>>
>>>>>>>>>> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/
>>>>>>>>>> linked-in:
>>>>>>>>>> http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>> Software Engineer, WSO2
>>>>>>>>>
>>>>>>>>> Mobile : +94777568639
>>>>>>>>> Blog : rajkumarr.com
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Rajkumar Rajaratnam
>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>> Software Engineer, WSO2
>>>>>>>>
>>>>>>>> Mobile : +94777568639
>>>>>>>> Blog : rajkumarr.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> --
>>>>>>> Lahiru Sandaruwan
>>>>>>> Committer and PMC member, Apache Stratos,
>>>>>>> Senior Software Engineer,
>>>>>>> WSO2 Inc., http://wso2.com
>>>>>>> lean.enterprise.middleware
>>>>>>>
>>>>>>> email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/
>>>>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Reka Thirunavukkarasu
>>>>>> Senior Software Engineer,
>>>>>> WSO2, Inc.:http://wso2.com,
>>>>>> Mobile: +94776442007
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Rajkumar Rajaratnam
>>>>> Committer & PMC Member, Apache Stratos
>>>>> Software Engineer, WSO2
>>>>>
>>>>> Mobile : +94777568639
>>>>> Blog : rajkumarr.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Reka Thirunavukkarasu
>>>> Senior Software Engineer,
>>>> WSO2, Inc.:http://wso2.com,
>>>> Mobile: +94776442007
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Rajkumar Rajaratnam
>>> Committer & PMC Member, Apache Stratos
>>> Software Engineer, WSO2
>>>
>>> Mobile : +94777568639
>>> Blog : rajkumarr.com
>>>
>>
>>
>>
>> --
>> Rajkumar Rajaratnam
>> Committer & PMC Member, Apache Stratos
>> Software Engineer, WSO2
>>
>> Mobile : +94777568639
>> Blog : rajkumarr.com
>>
>
>
>
> --
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
> Mobile: +94776442007
>
>
>
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
email: lahi...@wso2.com blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
