[
https://issues.apache.org/jira/browse/YARN-8250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510299#comment-16510299
]
Konstantinos Karanasos commented on YARN-8250:
----------------------------------------------
Just went through your comments. I am skeptical about adding a new
implementation of the Container Scheduler, as [~asuresh] and [~leftnoteasy]
also pointed out.
It seems we can achieve the required behavior just with some new
policies/functions rather than a new implementation.
>From what I understand, the main points are (1) how to start guaranteed
>containers (do we start them and then kill opportunistic, or do we kill
>opportunistic and then start them) and (2) how/when to start opportunistic
>containers (at regular intervals or when another container finishes).
*How to start guaranteed:*
For (1), just to make sure I understand: why can't we know the resources
utilized by each opportunistic container? But even if we don't know, can't we
just use the allocated resources of the container in our calculation? I
understand that this might lead to more opportunistic containers killed than
might be needed, but for sure it will be a good first version (and much better
than what we have today). At the same time, if we are not very aggressive with
starting opportunistic containers, it should not lead to too many containers
killed.
So my take is to have a first version that does the starting as it is today
(first queue, then kill), even if that leads to more opportunistic containers
killed. And be less aggressive in starting opportunistic for that matter.
*How to start opportunistic:*
Why not keep existing behavior and add the new one too? That is, we still let
opportunistic start when there is a container finish event, but do so not with
utilized but allocated resources. This will keep the current behavior. On top
of that, if over-allocation is enabled, we can have a thread that every so
often checks utilized resources and starts opportunistic containers on
over-allocated resources.
> Create another implementation of ContainerScheduler to support NM
> overallocation
> --------------------------------------------------------------------------------
>
> Key: YARN-8250
> URL: https://issues.apache.org/jira/browse/YARN-8250
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Haibo Chen
> Assignee: Haibo Chen
> Priority: Major
> Attachments: YARN-8250-YARN-1011.00.patch,
> YARN-8250-YARN-1011.01.patch, YARN-8250-YARN-1011.02.patch
>
>
> YARN-6675 adds NM over-allocation support by modifying the existing
> ContainerScheduler and providing a utilizationBased resource tracker.
> However, the implementation adds a lot of complexity to ContainerScheduler,
> and future tweak of over-allocation strategy based on how much containers
> have been launched is even more complicated.
> As such, this Jira proposes a new ContainerScheduler that always launch
> guaranteed containers immediately and queues opportunistic containers. It
> relies on a periodical check to launch opportunistic containers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
