[
https://issues.apache.org/jira/browse/YARN-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Babble Shack updated YARN-9737:
-------------------------------
Description:
Opportunistic scheduling is supposed to provide lower scheduling time, and thus
higher task throughput and lower job completion times for short jobs/tasks.
Through my experiments I have found distributed scheduling can degrade
performance.
I ran a gridmix trace of 100 short jobs, each with 50 tasks, with an average
run time of 1523ms.
Findings:
* Job completion time, the time take from submitting a job to job completion,
may degrade by over 200%
[^jct_cdf_100j_100t_1500.svg]
[^jct_cdf_100j_50t_1500_with_outliers.svg]
* Job execution time may increase by up to 300%
[^jet_boxplot_j100_50t_1500.svg]
[^jet_boxplot_j100_50t_1500_with_outliers.svg]
* Task throughput decreased by 100%
^[^task_throughput_boxplot_100j_50t_1500.svg]^
was:
Opportunistic scheduling is supposed to provide lower scheduling time, and thus
higher task throughput and lower job completion times for short jobs/tasks.
Through my experiments I have found distributed scheduling can degrade
performance.
I ran a gridmix trace of 100 short jobs, each with 50 tasks, with an average
run time of 1523ms.
Findings:
* Job completion time, the time take from submitting a job to job completion,
may degrade by over 200%
[^jct_cdf_100j_100t_1500.svg]
[^jct_cdf_100j_50t_1500_with_outliers.svg]
* Job execution time may increase by up to 300%
[^jet_boxplot_j100_50t_1500.svg]
[^jet_boxplot_j100_50t_1500_with_outliers.svg]
* Task throughput decreased by 400%
^[^task_throughput_boxplot_100j_50t_1500.svg]^
> Performance degradation, distributed opportunistic scheduling
> -------------------------------------------------------------
>
> Key: YARN-9737
> URL: https://issues.apache.org/jira/browse/YARN-9737
> Project: Hadoop YARN
> Issue Type: Bug
> Components: distributed-scheduling, yarn
> Affects Versions: 3.1.2
> Environment: OS: Ubuntu 18.04
> JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03
> 1 * Resource Manager – Intel Core i7-4770 CPU @ 3.40GHz, 16GB Memory, 256GB
> ssd.
> 37 * Node Managers - Intel Core i7-4770 CPU @ 3.40GHz, 8GB Memory, 256GB ssd.
> 2 * 3.5 Gb slots per Node Manager.
> yarn-site: [^yarn-site.xml]
> yarn-client-yarn-site: [^yarn-client.yarn-site.xml]
>
> Reporter: Babble Shack
> Priority: Major
> Attachments: jct_cdf_100j_100t_1500.svg,
> jct_cdf_100j_50t_1500_with_outliers.svg, jet_boxplot_j100_50t_1500.svg,
> jet_boxplot_j100_50t_1500_with_outliers.svg,
> task_throughput_boxplot_100j_50t_1500.svg, yarn-client.yarn-site.xml,
> yarn-site.xml
>
>
> Opportunistic scheduling is supposed to provide lower scheduling time, and
> thus higher task throughput and lower job completion times for short
> jobs/tasks.
> Through my experiments I have found distributed scheduling can degrade
> performance.
> I ran a gridmix trace of 100 short jobs, each with 50 tasks, with an average
> run time of 1523ms.
> Findings:
> * Job completion time, the time take from submitting a job to job
> completion, may degrade by over 200%
> [^jct_cdf_100j_100t_1500.svg]
> [^jct_cdf_100j_50t_1500_with_outliers.svg]
> * Job execution time may increase by up to 300%
> [^jet_boxplot_j100_50t_1500.svg]
> [^jet_boxplot_j100_50t_1500_with_outliers.svg]
> * Task throughput decreased by 100%
> ^[^task_throughput_boxplot_100j_50t_1500.svg]^
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
