On Tue, Mar 25, 2014 at 11:45:18PM -0400, Li Jin wrote: > Thanks for the reply. Still, it's not clear to me how DRF would help in > this case, let me elaborate: > > Let's say there are 3 frameworks A,B,C, running by user F1, F2, F3 and > there are 10 users, U1-U10, running tasks through A,B,C. > > Now use DRF between framework with equal weight, I believe the resource > will be equally distributed among the 3 frameworks. Is it possible for > Mesos to equally distribute the resource among the 10 users?
The simple answer is *NO*, DRF isn't a equal partition of resource, no one need equal partition in fact, it always depends on the resource request. The resource allocation is a dynamic progress, not a static partition for all available sources, thinking that there may be only a few users run tasks at the same time slot. For example, if there are more than one user and more than one tasks ready to run. Then, the DRF for user first selects user who has the minimum dominant resource share, if there are more than one tasks (say framework A task, framework B task, ...), then it selects the framework task which has the minimum dominant resource share. BTW, just my understanding from the paper, any mistake please correct me. -- Thanks, Chengwei > > Thanks, > Li > > > On Tue, Mar 25, 2014 at 10:39 PM, Chengwei Yang > <chengwei.yang...@gmail.com>wrote: > > > On Tue, Mar 25, 2014 at 06:17:11PM -0400, Li Jin wrote: > > > Dear Devs, > > > > > > We are seriously investigating using Mesos as the backbone of our compute > > > infrastructure. One important question I would like to ask is about fair > > > sharing. > > > > > > As I understand it, assuming you have 3 frameworks and 100 users using > > > those frameworks, the current algorithm gives each framework 33% > > (assuming > > > same weight), no matter how many users each framework have. In our case, > > > > I don't think so. By default, DRF allocator used among users and user's > > frameworks. > > > > See below options of mesos-master. > > > > --framework_sorter=VALUE Policy to use for allocating resources > > between a given user's frameworks. > > Options > > are the same as for user_allocator > > (default: drf) > > > > --user_sorter=VALUE Policy to use for allocating resources > > between users. May be one of: > > dominant_resource_fairness (drf) > > (default: drf) > > > > For DRF, please see this paper. > > http://people.csail.mit.edu/matei/papers/2011/nsdi_drf.pdf > > > > -- > > Thanks, > > Chengwei > > > > > actually we would like to give each user 1% of the cluster, no matter > > which > > > framework they use. The reasons are: > > > > > > (1) It's much easier for us to decide weight between users than weight > > > between framework. > > > (2) It makes it much easy to add and remove frameworks since it won't > > > change distribution of fair share > > > > > > In general, I feel frameworks compute on behave of users and thus users > > > should "pay" for the computation. > > > > > > I am wondering if this makes sense and if this is something could be > > > supported by Mesos. > > > > > > Thanks, > > > Li > >
