Thanks again for the reply. I should clarify I am considering the case where U1-U10 having unlimited requests. And I would like to give freed-up resource to the users (U1-U10) with the minimum drs instead of the framework (A,B,C) with mininum drs.
I am curious in how other people think and open to discussion. On Wed, Mar 26, 2014 at 12:12 AM, Chengwei Yang <[email protected]>wrote: > On Tue, Mar 25, 2014 at 11:45:18PM -0400, Li Jin wrote: > > Thanks for the reply. Still, it's not clear to me how DRF would help in > > this case, let me elaborate: > > > > Let's say there are 3 frameworks A,B,C, running by user F1, F2, F3 and > > there are 10 users, U1-U10, running tasks through A,B,C. > > > > Now use DRF between framework with equal weight, I believe the resource > > will be equally distributed among the 3 frameworks. Is it possible for > > Mesos to equally distribute the resource among the 10 users? > > The simple answer is *NO*, DRF isn't a equal partition of resource, no > one need equal partition in fact, it always depends on the resource > request. The resource allocation is a dynamic progress, not a static > partition for all available sources, thinking that there may be only a > few users run tasks at the same time slot. > > For example, if there are more than one user and more than one tasks > ready to run. Then, the DRF for user first selects user who has the > minimum dominant resource share, if there are more than one tasks (say > framework A task, framework B task, ...), then it selects the framework > task which has the minimum dominant resource share. > > BTW, just my understanding from the paper, any mistake please correct > me. > > -- > Thanks, > Chengwei > > > > > Thanks, > > Li > > > > > > On Tue, Mar 25, 2014 at 10:39 PM, Chengwei Yang > > <[email protected]>wrote: > > > > > On Tue, Mar 25, 2014 at 06:17:11PM -0400, Li Jin wrote: > > > > Dear Devs, > > > > > > > > We are seriously investigating using Mesos as the backbone of our > compute > > > > infrastructure. One important question I would like to ask is about > fair > > > > sharing. > > > > > > > > As I understand it, assuming you have 3 frameworks and 100 users > using > > > > those frameworks, the current algorithm gives each framework 33% > > > (assuming > > > > same weight), no matter how many users each framework have. In our > case, > > > > > > I don't think so. By default, DRF allocator used among users and user's > > > frameworks. > > > > > > See below options of mesos-master. > > > > > > --framework_sorter=VALUE Policy to use for allocating > resources > > > between a given user's frameworks. > > > Options > > > are the same as for user_allocator > > > (default: drf) > > > > > > --user_sorter=VALUE Policy to use for allocating > resources > > > between users. May be one of: > > > dominant_resource_fairness (drf) > > > (default: drf) > > > > > > For DRF, please see this paper. > > > http://people.csail.mit.edu/matei/papers/2011/nsdi_drf.pdf > > > > > > -- > > > Thanks, > > > Chengwei > > > > > > > actually we would like to give each user 1% of the cluster, no matter > > > which > > > > framework they use. The reasons are: > > > > > > > > (1) It's much easier for us to decide weight between users than > weight > > > > between framework. > > > > (2) It makes it much easy to add and remove frameworks since it won't > > > > change distribution of fair share > > > > > > > > In general, I feel frameworks compute on behave of users and thus > users > > > > should "pay" for the computation. > > > > > > > > I am wondering if this makes sense and if this is something could be > > > > supported by Mesos. > > > > > > > > Thanks, > > > > Li > > > >
