[jira] [Commented] (YARN-6443) Allow for Priority order relaxing in favor of improved node/rack locality
[
https://issues.apache.org/jira/browse/YARN-6443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15963202#comment-15963202
]
Arun Suresh commented on YARN-6443:
---
bq. Ah, so this apparently is describing a problem that only can occur if
scheduler keys are being used?
So, if all requests are made with different allocId, then yes, there is
performance degradation. But on the flip side, if no allocId were used, then
you might improve performance but at the cost of:
# No way of matching a returned container to a request
# No way of asking for different Resource sizes on the same Node.
I wouldn't say it is a bug per-se, since it is always possible for users to NOT
specify allocationId, and it would revert to the old behavior.
That said, I think we should change the heading of the JIRA. On further
analysis, we find that It is possible to improve performance without affecting
priority order. The confusion arose, as you pointed out, since we initially
included the allocId in the priority ordering, which we should not. But we
still need to fix the performance problem for users who want Containers matched
to requests.
For context, when we were initially prototyping the SchedulerKey feature, we
had considered pushing the allocationId down in the
{{AppSchedulingInfo::requests}} datastructure as follows:
1. Pre-allocationId state of the map:
{noformat}
requests = Map>
{noformat}
2. Initial Consideration:
{noformat}
requests = Map>>
{noformat}
3. Current Situation:
{noformat}
requests = Map>
where
SchdeulerKey = struct{Priority, AllocId}
{noformat}
2 and 3 above is actually somewhat equivalent, in terms of iteration cost
because you must still iterate over all the outstanding allocReqIds on a node
HB for the ANY case. For relaxLocality = false, 2 has a slight edge w.r.t 3
since the Requests for that node will be considered first, but in both cases,
all allocReqIds will be examined by the scheduler inner loop.
My understanding ([~hrsharma] can correct me) is that we can prune the list of
examined SchedulerKeys to consider only the Requests for the node and consider
the rack and ANY requests based on if the those SchedulerKeys have crossed the
missed opportunity threshold.
Will probably be easier to explain once we have a patch - which we will try to
attach here shortly.
> Allow for Priority order relaxing in favor of improved node/rack locality
> --
>
> Key: YARN-6443
> URL: https://issues.apache.org/jira/browse/YARN-6443
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler, fairscheduler
>Reporter: Arun Suresh
>Assignee: Hitesh Sharma
>
> Currently the Schedulers examine an applications pending Requests in Priority
> order. This JIRA proposes to introduce a flag (either via the
> ApplicationMasterService::registerApplication() or via some Scheduler
> configuration) to favor an ordering that is baised to the node that is
> currently heartbeating by relaxing the priority constraint.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6443) Allow for Priority order relaxing in favor of improved node/rack locality
[ https://issues.apache.org/jira/browse/YARN-6443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15960755#comment-15960755 ] Jason Lowe commented on YARN-6443: -- Ah, so this apparently is describing a problem that only can occur if scheduler keys are being used? I'm not sure we need a flag here. Seems like we simply should not guarantee that allocations are returned within a priority group in the order they are requested -- they can be returned in any order. It certainly worked that way without scheduler keys. If you need ordering, that's what priorities are for. In that sense I see this not as an enhancement but rather a bugfix. Or am I misunderstanding the problem? > Allow for Priority order relaxing in favor of improved node/rack locality > -- > > Key: YARN-6443 > URL: https://issues.apache.org/jira/browse/YARN-6443 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, fairscheduler >Reporter: Arun Suresh >Assignee: Hitesh Sharma > > Currently the Schedulers examine an applications pending Requests in Priority > order. This JIRA proposes to introduce a flag (either via the > ApplicationMasterService::registerApplication() or via some Scheduler > configuration) to favor an ordering that is baised to the node that is > currently heartbeating by relaxing the priority constraint. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6443) Allow for Priority order relaxing in favor of improved node/rack locality
[ https://issues.apache.org/jira/browse/YARN-6443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15960249#comment-15960249 ] Arun Suresh commented on YARN-6443: --- [~jlowe], Apologies for probably not wording this correctly. bq. If the app doesn't really care what order two different kinds of requests are given in as long as they have good locality, why not just submit them at the same priority instead of having this extra flag? This is essentially should solve the problem, but currently the allocationReqId is used as the secondary sorting key to order the SchedulerKeys. What we need is # Ignore the allocationId is the sort order # In the event that requests with the same Priority but different allocReqId are outstanding, we want a way to efficiently examine the key that matches the Node that is HB-ing first before going over the remaining keys. [~leftnoteasy], The patch you posted seemed to be doing something similar, but I guess that will work if there are multiple outstanding reqs againsts the same SchedulerKey. What we requires is efficient matching across schedulerKeys [~hrsharma], I understand you have a prototype for this - please do post it here. > Allow for Priority order relaxing in favor of improved node/rack locality > -- > > Key: YARN-6443 > URL: https://issues.apache.org/jira/browse/YARN-6443 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, fairscheduler >Reporter: Arun Suresh >Assignee: Hitesh Sharma > > Currently the Schedulers examine an applications pending Requests in Priority > order. This JIRA proposes to introduce a flag (either via the > ApplicationMasterService::registerApplication() or via some Scheduler > configuration) to favor an ordering that is baised to the node that is > currently heartbeating by relaxing the priority constraint. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6443) Allow for Priority order relaxing in favor of improved node/rack locality
[ https://issues.apache.org/jira/browse/YARN-6443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957308#comment-15957308 ] Arun Suresh commented on YARN-6443: --- (+ [~hrsharma] , [~roniburd] since they've been investigating very related stuff lately) > Allow for Priority order relaxing in favor of improved node/rack locality > -- > > Key: YARN-6443 > URL: https://issues.apache.org/jira/browse/YARN-6443 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, fairscheduler >Reporter: Arun Suresh >Assignee: Arun Suresh > > Currently the Schedulers examine an applications pending Requests in Priority > order. This JIRA proposes to introduce a flag (either via the > ApplicationMasterService::registerApplication() or via some Scheduler > configuration) to favor an ordering that is baised to the node that is > currently heartbeating by relaxing the priority constraint. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6443) Allow for Priority order relaxing in favor of improved node/rack locality
[ https://issues.apache.org/jira/browse/YARN-6443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957035#comment-15957035 ] Konstantinos Karanasos commented on YARN-6443: -- I was thinking about it last night and had the same concerns with Jason, i.e., couldn't think of cases when an application wants to "bypass" priority (which could lead to unexpected starvation of its tasks), instead of submitting tasks with the same priority and thereby give more weight to locality. > Allow for Priority order relaxing in favor of improved node/rack locality > -- > > Key: YARN-6443 > URL: https://issues.apache.org/jira/browse/YARN-6443 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, fairscheduler >Reporter: Arun Suresh >Assignee: Arun Suresh > > Currently the Schedulers examine an applications pending Requests in Priority > order. This JIRA proposes to introduce a flag (either via the > ApplicationMasterService::registerApplication() or via some Scheduler > configuration) to favor an ordering that is baised to the node that is > currently heartbeating by relaxing the priority constraint. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6443) Allow for Priority order relaxing in favor of improved node/rack locality
[ https://issues.apache.org/jira/browse/YARN-6443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956947#comment-15956947 ] Jason Lowe commented on YARN-6443: -- Could you elaborate a bit on the use case where an application bothers to specify different priorities for requests but is OK with having a lower priority request be allocated first because locality was better? Now that there are scheduler keys to help match allocations back to their original requests, it seems priority really can be a priority rather than a hack to help match them. If the app doesn't really care what order two different kinds of requests are given in as long as they have good locality, why not just submit them at the same priority instead of having this extra flag? > Allow for Priority order relaxing in favor of improved node/rack locality > -- > > Key: YARN-6443 > URL: https://issues.apache.org/jira/browse/YARN-6443 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, fairscheduler >Reporter: Arun Suresh >Assignee: Arun Suresh > > Currently the Schedulers examine an applications pending Requests in Priority > order. This JIRA proposes to introduce a flag (either via the > ApplicationMasterService::registerApplication() or via some Scheduler > configuration) to favor an ordering that is baised to the node that is > currently heartbeating by relaxing the priority constraint. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6443) Allow for Priority order relaxing in favor of improved node/rack locality
[ https://issues.apache.org/jira/browse/YARN-6443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956325#comment-15956325 ] Wangda Tan commented on YARN-6443: -- [~arun.sur...@gmail.com], I was thinking about similar things while working on YARN-5139. One issue I can think of is, it will break ordering of applications/queues/schedulerKeys. The ideal solution in my mind is pass multiple nodes while doing scheduling and scoring/sorting nodes based on placement policy (for example, based on locality). There're some optimizations we can make to reduce the overhead of sorting. I put some of the stuffs in early patch of YARN-5139: https://issues.apache.org/jira/secure/attachment/12825346/wip-4.YARN-5139.patch#file-36 please check LocalityNodesScorer.java. I think we can use the similar approach to optimize affinity/anti-affinity which I mentioned offline. Please let me know your thoughts. > Allow for Priority order relaxing in favor of improved node/rack locality > -- > > Key: YARN-6443 > URL: https://issues.apache.org/jira/browse/YARN-6443 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, fairscheduler >Reporter: Arun Suresh >Assignee: Arun Suresh > > Currently the Schedulers examine an applications pending Requests in Priority > order. This JIRA proposes to introduce a flag (either via the > ApplicationMasterService::registerApplication() or via some Scheduler > configuration) to favor an ordering that is baised to the node that is > currently heartbeating by relaxing the priority constraint. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6443) Allow for Priority order relaxing in favor of improved node/rack locality
[ https://issues.apache.org/jira/browse/YARN-6443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956309#comment-15956309 ] Arun Suresh commented on YARN-6443: --- My initial though was: instead of simply iterating over appschedingInfo::getSchedulerKeys(), we have a Policy that takes the current Node and returns an iterator with SchedulerKeys that have ResourceRequests with the current node as ResourceName bubble to the top. [~jlowe], [~leftnoteasy], [~kasha], [~vinodkv], [~curino], [~kkaranasos] Thoughts ? > Allow for Priority order relaxing in favor of improved node/rack locality > -- > > Key: YARN-6443 > URL: https://issues.apache.org/jira/browse/YARN-6443 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, fairscheduler >Reporter: Arun Suresh >Assignee: Arun Suresh > > Currently the Schedulers examine an applications pending Requests in Priority > order. This JIRA proposes to introduce a flag (either via the > ApplicationMasterService::registerApplication() or via some Scheduler > configuration) to favor an ordering that is baised to the node that is > currently heartbeating by relaxing the priority constraint. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
