[
https://issues.apache.org/jira/browse/YARN-10174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766648#comment-17766648
]
Shilun Fan edited comment on YARN-10174 at 9/19/23 5:51 AM:
------------------------------------------------------------
Thank you very much for bringing up this JIRA. Here's my understanding, and I
look forward to further discussing it with you:
Currently, YARN Federation supports the allocation and scheduling of jobs to
various subclusters based on Router Policy and AMRM Policy.
For example:
Consider the queue *root.a*, which is associated with three subclusters:
`SubCluster-1`, `SubCluster-2`, and `SubCluster-3`. In our current setup, the
Router Policy is defined as `SubCluster-1:0.2, SubCluster-2:0.5,
SubCluster-3:0.3`, and the AMRM Policy is `SubCluster-1:0.3, SubCluster-2:0.4,
SubCluster-3:0.3`. This is a fixed configuration.
However, we may have various types of jobs, such as AdHoc jobs or SLA jobs, and
the predefined rules may not necessarily align well with the specific needs of
these jobs. Therefore, when submitting jobs from the client side, it would be
beneficial to express job preferences.
In this JIRA, we will be adding support for configuring multiple weight
combinations for a specific queue. For example:
Router Policy:
SLA: `SubCluster-1:1, SubCluster-2:0, SubCluster-3:0`
AdHoc: `SubCluster-1:0.2, SubCluster-2:0.5, SubCluster-3:0.3`
Default: `SubCluster-1:0.2, SubCluster-2:0.5, SubCluster-3:0.3`
When clients submit tasks, they can specify their task type in the tag.
was (Author: slfan1989):
Thank you very much for bringing up this JIRA. Here's my understanding, and I
look forward to further discussing it with you:
Currently, YARN Federation supports the allocation and scheduling of jobs to
various subclusters based on Router Policy and AMRM Policy.
For example:
Consider the queue *root.a*, which is associated with three subclusters:
`SC-1`, `SC-2`, and `SC-3`. In our current setup, the Router Policy is defined
as `SC-1:0.2, SC-2:0.5, SC-3:0.3`, and the AMRM Policy is `SC-1:0.3, SC-2:0.4,
SC-3:0.3`. This is a fixed configuration.
However, we may have various types of jobs, such as AdHoc jobs or SLA jobs, and
the predefined rules may not necessarily align well with the specific needs of
these jobs. Therefore, when submitting jobs from the client side, it would be
beneficial to express job preferences.
In this JIRA, we will be adding support for configuring multiple weight
combinations for a specific queue. For example:
Router Policy:
SLA: `SC-1:1, SC-2:0, SC-3:0`
AdHoc: `SC-1:0.2, SC-2:0.5, SC-3:0.3`
Default: `SC-1:0.2, SC-2:0.5, SC-3:0.3`
When clients submit tasks, they can specify their task type in the tag.
> Add colored policies to enable manual load balancing across sub clusters
> ------------------------------------------------------------------------
>
> Key: YARN-10174
> URL: https://issues.apache.org/jira/browse/YARN-10174
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Young Chen
> Assignee: zhengchenyu
> Priority: Major
>
> Add colored policies to enable manual load balancing across sub clusters
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
