[
https://issues.apache.org/jira/browse/YARN-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933869#comment-16933869
]
Konstantinos Karanasos commented on YARN-9737:
----------------------------------------------
Hi [~Babbleshack], just saw this.
The performance of opportunistic containers depends on a lot of things -- here
are a few to consider:
* Note that distributed scheduling is most probably not what affects you here,
but instead the use of opportunistic containers. So, I am pretty sure you would
get the same results for centralized scheduling of opportunistic containers.
* Opportunistic containers can be killed by guaranteed containers, therefore
execution is much more sensitive as cluster utilization increases. So if your
gridmix script ends up leading to high cluster utilization, you might end up
getting excessive killing or queuing, hence the slowdown you observe.
* Along the same lines, you might want to decrease the number of concurrent
applications, especially if your utilization is high. Opportunistic containers
do not count towards actually used resources for that matter, so if you are not
careful you will end up launching too many jobs, therefore too many AMs, which
use guaranteed containers and will be killing running opportunistic ones.
* The fact that you allow at most two containers per node does not help either
(and it is not a very common set-up in practice). It means that when a
guaranteed container arrives at that node, at least 50% of the node's
opportunistic containers will be killed (just by killing a single container).
* You might also try to see what happens if you decrease the size of your
queue to 0 or 1 (you will not avoid killing but you will avoid queuing).
* How are you setting the percentage of opportunistic containers? I guess in
the AM? Note that even when you set it to 100%, the AM will still be launched
with a guaranteed container.
Hope this helps. Then there are a lot of other things to tune for the
scheduling of opportunistic containers, but I would start from the above list.
cc: [~abmodi] that is currently working on opportunistic containers actively.
> Performance degradation, Distributed Opportunistic Scheduling
> -------------------------------------------------------------
>
> Key: YARN-9737
> URL: https://issues.apache.org/jira/browse/YARN-9737
> Project: Hadoop YARN
> Issue Type: Bug
> Components: distributed-scheduling, yarn
> Affects Versions: 3.1.2
> Environment: OS: Ubuntu 18.04
> JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03
> 1 * Resource Manager – Intel Core i7-4770 CPU @ 3.40GHz, 16GB Memory, 256GB
> ssd.
> 37 * Node Managers - Intel Core i7-4770 CPU @ 3.40GHz, 8GB Memory, 256GB
> ssd.
> 2 * 3.5 Gb slots per Node Manager, 1x cpu per slot
> yarn-site: [^yarn-site.xml]
> yarn-client-yarn-site: [^yarn-client.yarn-site.xml]
>
> Reporter: Babble Shack
> Priority: Major
> Labels: performance, scheduler, scheduling
> Attachments: jct_cdf_100j_100t_1500.svg,
> jct_cdf_100j_50t_1500_with_outliers.svg, jet_boxplot_j100_50t_1500.svg,
> jet_boxplot_j100_50t_1500_with_outliers.svg,
> task_throughput_boxplot_100j_50t_1500.svg, yarn-client.yarn-site.xml,
> yarn-site.xml
>
>
> Opportunistic scheduling is supposed to provide lower scheduling time, and
> thus higher task throughput and lower job completion times for short
> jobs/tasks.
> Through my experiments I have found distributed scheduling can degrade
> performance.
> I ran a gridmix trace of 100 short jobs, each with 50 tasks. Average task run
> time was 1523ms.
> Findings:
> * Job completion time, the time take from submitting a job to job
> completion, may degrade by over 200%
> [^jct_cdf_100j_100t_1500.svg]
> [^jct_cdf_100j_50t_1500_with_outliers.svg]
> * Job execution time may increase by up to 300%
> [^jet_boxplot_j100_50t_1500.svg]
> [^jet_boxplot_j100_50t_1500_with_outliers.svg]
> * Task throughput decreased by 100%
> ^[^task_throughput_boxplot_100j_50t_1500.svg]^
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
