[
https://issues.apache.org/jira/browse/YARN-6808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086186#comment-16086186
]
Wangda Tan commented on YARN-6808:
----------------------------------
[~asuresh], thanks for the detailed explanations.
I can understand there're two separate targets, but I'm not sure how the 2
targets related to each other.
1) Use opportunistic container to do lazy preemption in NM. (Is there any
umbrella JIRA for this?)
2) Convert guaranteed request to opportunistic request when app's headroom
reached.
Questions:
1) Let's say app1 in an underutilized queue, which want to preempt containers
from an over-utilized queue. Will preemption happens if app1 asks opportunistic
container?
2) For target #1, who make the decision of moving guaranteed containers to
opportunistic containers. If it is still decided by central RM, does that mean
preemption logics in RM are same as today except kill operation is decided by
NM side?
3) For overall opportunistic container execution: If OC launch request will be
queued by NM, it may wait a long time before get executed. In this case, do we
need to modify AM code to: a. expect longer delay before think the launch
fails. b. asks more resource on different hosts since there's no guaranteed
launch time for OC?
Comments for target #2.
- What happens if an app doesn't want to ask opportunistic container when go
beyond headroom? (Such as online services). I think this should be a per-app
config (give me OC when I'm go beyond headroom).
- Existing patch makes static decision, which happens when new resource request
added by AM. Should this be reconsidered when app's headroom changed over time?
Overall, I think this is a big feature and involves lots of components.
Including a more detailed design doc can help contributors understand its scope
and workflow.
> Allow Schedulers to return OPPORTUNISTIC containers when queues go over
> configured capacity
> -------------------------------------------------------------------------------------------
>
> Key: YARN-6808
> URL: https://issues.apache.org/jira/browse/YARN-6808
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Arun Suresh
> Assignee: Arun Suresh
> Attachments: YARN-6808.001.patch
>
>
> This is based on discussions with [~kasha] and [~kkaranasos].
> Currently, when a Queues goes over capacity, apps on starved queues must wait
> either for containers to complete or for them to be pre-empted by the
> scheduler to get resources.
> This JIRA proposes to allow Schedulers to:
> # Allocate all containers over the configured queue capacity/weight as
> OPPORTUNISTIC.
> # Auto-promote running OPPORTUNISTIC containers of apps as and when their
> GUARANTEED containers complete.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
