[
https://issues.apache.org/jira/browse/YARN-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14042469#comment-14042469
]
Craig Welch commented on YARN-1039:
-----------------------------------
{quote}We need to have a flag to indicate an AM is long lived. We need to have
another flag to indicate that the containers requested by an AM will be long
lived,{quote}
I believe the proposal meets those needs, it is one particular way to do so...
{quote}it must be set at application creation time and all containers of the
app will be considered long lived. This is because the RM does not keep track
of individual container requests.{quote}
I'm not sure why it matters whether or not RM keeps track of the container
requests - the AM will request containers with scheduling constraints like long
lived, affinity, etc, and RM considers them when selecting nodes, after that
completes it no longer necessarily matters. If the AM needs a relationship
between nodes for a request, or a particular type of selection (not on a spot
node - long-running) it will make a request for those nodes, get nodes that
meet it's needs, and it's good to go. It sounds as though it would be more
flexible / meet a wider set of usecases and therefore be preferable to allow an
application master to obtain different types of containers for different
purposes during it's lifetime as opposed to forcing to use only one set of
container constraints throughout
{quote}Having a long enum of flag to indicated optional qualities of the
requested containers has been discussed in the past (in the context of some
JIRAs related to Llama) and it has been discarded as it would mean divergence
on the features different schedulers support.{quote}
So, for this jira there is a desire to support selecting nodes with particular
qualities (not placing a long running process on a temporary/spot instance),
coming soon are other needs for other similar selection/constraint logic
(affinity, anti-affinity, etc) - not being able to indicate qualities for the
containers would keep us for being able to support those needs, and I believe
there is a need to support this functionality. It's
filtering/constraint/selection logic and could probably be generalized in a way
which could be used by various schedulers...
> Add parameter for YARN resource requests to indicate "long lived"
> -----------------------------------------------------------------
>
> Key: YARN-1039
> URL: https://issues.apache.org/jira/browse/YARN-1039
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Affects Versions: 3.0.0, 2.1.1-beta
> Reporter: Steve Loughran
> Assignee: Craig Welch
> Attachments: YARN-1039.1.patch, YARN-1039.2.patch, YARN-1039.3.patch
>
>
> A container request could support a new parameter "long-lived". This could be
> used by a scheduler that would know not to host the service on a transient
> (cloud: spot priced) node.
> Schedulers could also decide whether or not to allocate multiple long-lived
> containers on the same node
--
This message was sent by Atlassian JIRA
(v6.2#6252)
