[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095739#comment-15095739
]
Klaus Ma commented on MESOS-3765:
-
Added link of design doc; this's the draft version. We're working on it.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>Assignee: Guangya Liu
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095733#comment-15095733
]
John White commented on MESOS-3765:
---
[~gyliu] [~fan.du], can you guys post the link of the design doc here so that
more folks can review and comment?
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>Assignee: Guangya Liu
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095742#comment-15095742
]
Guangya Liu commented on MESOS-3765:
[~jkw], it is in very early stage, will publish it out when we get something
done. Hope it is OK, thanks.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>Assignee: Guangya Liu
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093477#comment-15093477
]
Fan Du commented on MESOS-3765:
---
Sure, will do.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>Assignee: Guangya Liu
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093455#comment-15093455
]
Fan Du commented on MESOS-3765:
---
[~gyliu] The proposal document states "DRF will be disabled with Fine-Grained
Resource Offers." , I am wondering why fine grained offer should bypass WDRF in
practice?
By my understanding, impliments fine grained offer fits well inside current
WDRF logic, because of current allocation behavior:
Foreach Slave
Foreach Role
Foreach Framework within the role
compute agent resources of revocable case OR
compute agent resources of non-revocable case <- (*A)
offer the agent resources to current framework <- (*B)
Each slave will grant at most one time allocation offer for the first framework
within a role, if there is no revocalbe frameworks;
Each slave will grant at most two times allocations offer for one non-revocable
and one revocalbe framework.
If we apply granuality between (*A) and (*B), it would be perfet to make loops
to iterate remaining framworks,
the the goal to spread agent resource between frameworks is done.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>Assignee: Guangya Liu
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093467#comment-15093467
]
Guangya Liu commented on MESOS-3765:
Hi [~fan.du] you can update the document directly by adding some of your
thinking there, we can iterate to work on the document. The document is just
some very early thinking, yes, we can still leverage DRF even with fine grained
resource offer.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>Assignee: Guangya Liu
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993467#comment-14993467
]
Guangya Liu commented on MESOS-3765:
[~alexr] do you mean that you and [~hekaldama] will work on the design
document? Can you please share some content of the IRC discussion between you
and [~hekaldama]? How does {{--resource}} and {{reservation}} help on this?
Thanks!
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993690#comment-14993690
]
Alexander Rukletsov commented on MESOS-3765:
There is one more application of this in regarding of quota. If the unsatisfied
part of quota is, say, {{1 cpus}}, the next agent we fetch from the queue has
{{10 cpus}} available, we will offer {{10 cpus}} as part of quota, though we
may want to offer just {{1}}.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>Assignee: Guangya Liu
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993679#comment-14993679
]
Qian Zhang commented on MESOS-3765:
---
Thanks [~adam-mesos], actually I am reviewing the patches of quota epic, but I
do not see how quota can be subdivided at least in the MVP, maybe what you are
talking about is chunk which we will implement it in the post-MVP?
And for weight, if we handle it in the way that you mentioned above, I think
actually we would have changed its original meaning / purpose, originally
weight is kind of share priority, but now we will want it to represent share
ratio, right?
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>Assignee: Guangya Liu
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993553#comment-14993553
]
Adam B commented on MESOS-3765:
---
The IRC discussion with Adam Avilla turned out to be unrelated to this ticket.
He needed a way to exclude resources from Mesos' allocations, which is solved
by not including them in `--resources` if the tasks are run outside of Mesos,
and can be solved by reservations for tasks run by Mesos.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993570#comment-14993570
]
Guangya Liu commented on MESOS-3765:
Thanks [~adam-mesos] I will take over this task and do some test.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993554#comment-14993554
]
Adam B commented on MESOS-3765:
---
The IRC discussion with Adam Avilla turned out to be unrelated to this ticket.
He needed a way to exclude resources from Mesos' allocations, which is solved
by not including them in `--resources` if the tasks are run outside of Mesos,
and can be solved by reservations for tasks run by Mesos.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993617#comment-14993617
]
Qian Zhang commented on MESOS-3765:
---
[~adam-mesos], I am just curious how master/allocator knows a framework's ideal
fair share, can you please explain a bit more about that?
And will we also consider weight? E.g., if role1's weight is larger than
role2's weight, then role1's framework will get larger offer than role2's
framework. Maybe send offer in proportion to role's weight?
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>Assignee: Guangya Liu
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993646#comment-14993646
]
Adam B commented on MESOS-3765:
---
We would definitely consider role weights. A role's fair share is calculated
based on the number of active roles (with registered frameworks) and their
weights. With 4 roles each with a weight of 1, each role's fair share of the
cluster is 25% of the total cpu/mem/disk resources. If one of those roles has 5
frameworks in it, then each of those frameworks' fair share is 5% of the
cluster resources. (There are no per-framework weights.)
Quota would be global rather than per-node, but even with a global quota, it
may be subdivided so that a framework can say "I need 5 instances of {2cpu, 4GB
RAM}, although I don't care which nodes those instances come from." As such, we
would use {2cpu, 4GB RAM} as the minimum per-node offer to a framework, rather
than offering the entire node. See MESOS-1791 for more information about quota.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>Assignee: Guangya Liu
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992716#comment-14992716
]
Adam Avilla commented on MESOS-3765:
+1
I know we have talked about wanting this feature namely in the context of
agents need to run other things and can't be fully consumed by mesos tasks only.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992741#comment-14992741
]
Alexander Rukletsov commented on MESOS-3765:
As per our discussion on IRC, {{--resources}} flag should help in providing
headroom for out-of-band apps like system daemons. For mesos tasks that should
run on particular agents, reservations (both dynamic and static) can be used.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992738#comment-14992738
]
Alexander Rukletsov commented on MESOS-3765:
I think we can start writing a design doc for this change with the focus on
quota. I can help guide the design doc, and once the design is settled, we can
try to solicit a shepherd.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992744#comment-14992744
]
Klaus Ma commented on MESOS-3765:
-
Let me draft a design document for it :).
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>Assignee: Klaus Ma
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992743#comment-14992743
]
Klaus Ma commented on MESOS-3765:
-
Let me draft a design document for it :).
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>Assignee: Klaus Ma
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972772#comment-14972772
]
Alexander Rukletsov commented on MESOS-3765:
Yep, that's what I have in mind.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972957#comment-14972957
]
Guangya Liu commented on MESOS-3765:
[~alexr] I think that your original thought is enable cluster level, when start
up mesos master, the cluster operator can define the "granularity" or
"allocation unit" for all of the frameworks.
After a second thought, I think that the "granularity" should be defined by the
framework as only the framework knows how many resource does he want, each
framework can just define its "granularity" or "allocation unit" to its task
size (or else we can treat the task size as allocation unit) and request
resource to Mesos via `requestResources()`, the mesos master will help
calculate if the requirement can be satisfied and launch task accordingly. I
think that this is the best way to achieve the "granularity".
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972958#comment-14972958
]
Guangya Liu commented on MESOS-3765:
[~alexr] I think that your original thought is enable cluster level, when start
up mesos master, the cluster operator can define the "granularity" or
"allocation unit" for all of the frameworks.
After a second thought, I think that the "granularity" should be defined by the
framework as only the framework knows how many resource does he want, each
framework can just define its "granularity" or "allocation unit" to its task
size (or else we can treat the task size as allocation unit) and request
resource to Mesos via `requestResources()`, the mesos master will help
calculate if the requirement can be satisfied and launch task accordingly. I
think that this is the best way to achieve the "granularity".
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14973028#comment-14973028
]
Alexander Rukletsov commented on MESOS-3765:
Let's briefly touch that during the next sync. I would like to figure out
whether it makes sense to jump on that before we refactor the allocator. Also
which working group should take care of this and who will shepherd.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14973044#comment-14973044
]
Klaus Ma commented on MESOS-3765:
-
Make sense to me, let's review this at next sync meeting.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972953#comment-14972953
]
Klaus Ma commented on MESOS-3765:
-
OK, copy that. [~alexr], would you shepherd on this? Let me draft a design
document.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972948#comment-14972948
]
Qian Zhang commented on MESOS-3765:
---
I think granularity is just a way to divide the entire agent resources into
small pieces, so that *it is possible* for allocator to allocate resources in a
more fine-grained mode, but that will not render the fairness automatically. I
think the key is how allocator offers resources to frameworks based on the
granularity in each allocation cycle, i.e., the algorithm to decide how many
pieces (granularity) should be offered to each framework. [~alexr], can we have
a design doc for this improvement?
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971016#comment-14971016
]
Klaus Ma commented on MESOS-3765:
-
Thanks for explaining current behaviour, it match my understanding :).
According to the description of this ticket, we consider assigning all
resources of slave to framework is unfair, right? So which case is fairness by
re-using the example?
Two framework {{f1}} and {{f2}}, one agent with only 1 CPU:
1. 0.5 to {{f1}} and 0.5 to {{f2}} (both weight are 1: {{total * weight/ sum
of weight}})
2. 1 to {{f1}} and 0 to {{f2}}
3. or others?
IMO, #1 is fair to both framework; but if {{f1}}/{{f2}} acquired 1 CPU to
launch task, it need a way for allocator to adjust the fairness, so
{{requestResources()}} will help.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970790#comment-14970790
]
Alexander Rukletsov commented on MESOS-3765:
This should work, but changes the current behaviour a bit. Let me reuse your
example to demonstrate how it works right now.
Two frameworks {{f1}} and {{f2}}, one agent with only 1 CPU. Possible scenarios:
* {{f1}} (or {{f2}}) receives 1 CPU, but accepts 0.5 CPU, effectively
returning 0.5 CPU to the free pool. {{f2}} is offered 0.5 CPU.
* {{f1}} (or {{f2}}) is greedy and accepts 1 CPU, rendering {{f2}} starve.
If a framework is a good citizen, it will accept only as much resources, as it
needs. At the same time, if a framework is good citizen, it won't lie about its
granularity. Hence I don't really see how having frameworks report their
preferable offer size can help to resolve allocation unfairness (which is
described in the ticket). I do not argue having frameworks' preferable offer
size allows for smarter allocation decisions and more efficient bin packing
algorithms, but this is out of the scope of the ticket.
My intuition is that *there is no way to improve fairness by giving frameworks
more tuning mechanism*. Having said that, I think tuning mechanism like
{{requestResources}} can be extremely useful.
Also, I think it can be difficult to deduce default granularity in presence of
multiple frameworks and varying agents.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971282#comment-14971282
]
Alexander Rukletsov commented on MESOS-3765:
My answer is: it depends. I would like to give an operator the ability to
choose what is better for their cluster.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972219#comment-14972219
]
Klaus Ma commented on MESOS-3765:
-
So the "granularity" is cluster level and access (CRUD) only by operator. In
allocator, it will assign resources by granularity instead of all slave
resources.
And I think the granularity should have min value, or the allocation will be
slow :).
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970104#comment-14970104
]
Klaus Ma commented on MESOS-3765:
-
But how to define the "allocation chunk"? For example, two framework(f1 and
f2), each of them have a task acquire 1 CPU; one slave with only 1 CPU. I think
we need a way to let allocator know the minimal resource requirement: by
{{requestResources()}} or by {{filter}}. The framework can NOT define the
resource requirement ahead because it dependent on the pending workload in it.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969288#comment-14969288
]
Klaus Ma commented on MESOS-3765:
-
For [~gyliu]'s proposal on {{requestResources}}, I think that would be a long
term solution for us; maybe we can re-visit it in "allocator refactor" ticket.
[~alexr], for the "granularity", suggest to make the default value to be
{{total resource/framework #}} of each resources. But one concern is if lots of
frameworks, the allocator's performance will be downgrade. Maybe we can have a
try to see the performance. Any comments.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969382#comment-14969382
]
Klaus Ma commented on MESOS-3765:
-
[~alexr], I think we can use {{requestResources()}} to send granularity; so
allocator send offer based on granularity in each framework (the granularity
maybe different within one framework). Here's a workflow in allocator:
1. if there's no available resources to allocate, return
2. for each framework, allocate "granularity" resources in each loop, the
"granularity" will be a.) "total/framework #" for each resource type
(cput/mem), if no "granularity" from framework; b.) the "granularity" from
framework sent by {{requestResources()}}
3. keep running #2 until no available resources.
The frame work will fair-share the resource in the cluster, although greedy
framework may send several "granularity" to allocator; because allocator will
give only one "granularity" to framework in each loop. There is still a case
that framework overcommit by "granularity" (granularity > required resources )
which we can not guarantee, because allocator did not know framework's task
size.
*Demonstration:*
Two framework(f1 and f2), each of them will acquire 1 CPU; one slave with only
1 CPU; so the workflow will be:
- each framework will receive 0.5 CPU because no "granularity" from
framework, the default "granularity" is 1 CPU /2 framework.
- each framework will send "granularity" by {{requestResources}} for 1 CPU,
because the offer can not meet the requirement (1 vs. 0.5)
- allocator will send the 1 CPU to f1 (or f2) based on its "granularity"
One concern is performance downgrade, we need a perf test for it; and maybe a
flag to control this behaviour.
If any comments, please let me know.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965176#comment-14965176
]
Alexander Rukletsov commented on MESOS-3765:
Yes, that's what I have in mind. I would avoid calling that an offer, it's
rather an allocation chunk. The allocator may still allocate multiple chunks to
a single framework in one allocation cycle, which will end up in a single offer.
An alternative is percentage, but we should still stick to the original agent
size, as taking a fraction of the remaining resources on a agent can be a very
small value.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965189#comment-14965189
]
Guangya Liu commented on MESOS-3765:
There might be problems that the master/allocator do not know the exact
resource request of the framework, so it seems difficult to let
master/allocator satisfy the request of the framework and sometimes this may
cause the framework starve if the offer do not have enough resources.
Mesos now support requestResource, can we leverage this API? The framework can
just send the exact resource request to Mesos master and the master can return
the offer with the exact request resource to framework, comments?
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965213#comment-14965213
]
Alexander Rukletsov commented on MESOS-3765:
{quote}
There might be problems that the master/allocator do not know the exact
resource request of the framework, so it seems difficult to let
master/allocator satisfy the request of the framework
{quote}
But this is true for the status quo, right? Currently the allocator does not
take into consideration frameworks resource wishes, if any. This ticket
proposes to make the "allocation chunk" adjustable. IIUC, your proposal is to
implement {{requestResources()}}, which is in my opinion is a separate
discussion. Also note that a framework may hoard resources, which means having
multiple smaller chunks should not be a big problem.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965245#comment-14965245
]
Guangya Liu commented on MESOS-3765:
I think that the final goal is "fine-grained" resource scheduling and enabling
the master/allocator know the exact resource request is a easy way to implement
this. ;-)
Another point is that a framework hoard resources, do you mean the the
allocator can use filter to hoard resources for a framework? But the problem is
that the current filter is host level and even after filter expires, the
allocator still using the allocation unit to allocate resource offers and this
may still cannot satisfy the framework request. With "granularity", we may need
to set the filter to allocation unit level but not host level.
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3765) Make offer size adjustable (granularity)
[
https://issues.apache.org/jira/browse/MESOS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965104#comment-14965104
]
Guangya Liu commented on MESOS-3765:
[~alexr] Does the "granularity" is kind of allocation unit, such as
"cpu:1;mem:256", then the allocator will treat "cpu:1;mem:256" as one offer?
> Make offer size adjustable (granularity)
>
>
> Key: MESOS-3765
> URL: https://issues.apache.org/jira/browse/MESOS-3765
> Project: Mesos
> Issue Type: Improvement
> Components: allocation
>Reporter: Alexander Rukletsov
>
> The built-in allocator performs "coarse-grained" allocation, meaning that it
> always allocates the entire remaining agent resources to a single framework.
> This may heavily impact allocation fairness in some cases, for example in
> presence of numerous greedy frameworks and a small number of powerful agents.
> A possible solution would be to allow operators explicitly specify
> granularity via allocator flags. While this can be tricky for non-standard
> resources, it's pretty straightforward for {{cpus}} and {{mem}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
