[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[ https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088457#comment-17088457 ] László Bodor commented on HIVE-23210: - pushed to master, thanks [~pgaref] for the patch and [~rajesh.balamohan] for the review! > Fix shortestjobcomparator when jobs submitted have 1 task their vertices > > > Key: HIVE-23210 > URL: https://issues.apache.org/jira/browse/HIVE-23210 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23210.01.patch, HIVE-23210.02.patch, > HIVE-23210.03.patch, HIVE-23210.04.patch, HIVE-23210.wip.patch, > TestShortestJobFirstComparator.java > > Time Spent: 10m > Remaining Estimate: 0h > > In latency sensitive queries, lots of jobs can have vertices with 1 task. > Currently shortestjobcomparator does not work correctly and returns tasks in > random order. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51] > This causes delay in the job runtime. I will attach a simple test case > shortly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[ https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087599#comment-17087599 ] Rajesh Balamohan commented on HIVE-23210: - Thanks [~pgaref] . Latest patch LGTM. +1. > Fix shortestjobcomparator when jobs submitted have 1 task their vertices > > > Key: HIVE-23210 > URL: https://issues.apache.org/jira/browse/HIVE-23210 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23210.01.patch, HIVE-23210.02.patch, > HIVE-23210.03.patch, HIVE-23210.04.patch, HIVE-23210.wip.patch, > TestShortestJobFirstComparator.java > > Time Spent: 10m > Remaining Estimate: 0h > > In latency sensitive queries, lots of jobs can have vertices with 1 task. > Currently shortestjobcomparator does not work correctly and returns tasks in > random order. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51] > This causes delay in the job runtime. I will attach a simple test case > shortly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[
https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086730#comment-17086730
]
Hive QA commented on HIVE-23210:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13000390/HIVE-23210.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17144 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21762/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21762/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21762/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13000390 - PreCommit-HIVE-Build
> Fix shortestjobcomparator when jobs submitted have 1 task their vertices
>
>
> Key: HIVE-23210
> URL: https://issues.apache.org/jira/browse/HIVE-23210
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Panagiotis Garefalakis
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23210.01.patch, HIVE-23210.02.patch,
> HIVE-23210.03.patch, HIVE-23210.04.patch, HIVE-23210.wip.patch,
> TestShortestJobFirstComparator.java
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In latency sensitive queries, lots of jobs can have vertices with 1 task.
> Currently shortestjobcomparator does not work correctly and returns tasks in
> random order.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51]
> This causes delay in the job runtime. I will attach a simple test case
> shortly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[
https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086726#comment-17086726
]
Hive QA commented on HIVE-23210:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} llap-server in master has 89 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} llap-server: The patch generated 2 new + 13 unchanged
- 0 fixed = 15 total (was 13) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21762/dev-support/hive-personality.sh
|
| git revision | master / c3fb689 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21762/yetus/diff-checkstyle-llap-server.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21762/yetus/patch-asflicense-problems.txt
|
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21762/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix shortestjobcomparator when jobs submitted have 1 task their vertices
>
>
> Key: HIVE-23210
> URL: https://issues.apache.org/jira/browse/HIVE-23210
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Panagiotis Garefalakis
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23210.01.patch, HIVE-23210.02.patch,
> HIVE-23210.03.patch, HIVE-23210.04.patch, HIVE-23210.wip.patch,
> TestShortestJobFirstComparator.java
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In latency sensitive queries, lots of jobs can have vertices with 1 task.
> Currently shortestjobcomparator does not work correctly and returns tasks in
> random order.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51]
> This causes delay in the job runtime. I will attach a simple test case
> shortly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[
https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086638#comment-17086638
]
Hive QA commented on HIVE-23210:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13000372/HIVE-23210.03.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17144 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query20]
(batchId=226)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21756/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21756/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21756/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13000372 - PreCommit-HIVE-Build
> Fix shortestjobcomparator when jobs submitted have 1 task their vertices
>
>
> Key: HIVE-23210
> URL: https://issues.apache.org/jira/browse/HIVE-23210
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Panagiotis Garefalakis
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23210.01.patch, HIVE-23210.02.patch,
> HIVE-23210.03.patch, HIVE-23210.wip.patch, TestShortestJobFirstComparator.java
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In latency sensitive queries, lots of jobs can have vertices with 1 task.
> Currently shortestjobcomparator does not work correctly and returns tasks in
> random order.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51]
> This causes delay in the job runtime. I will attach a simple test case
> shortly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[
https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086630#comment-17086630
]
Hive QA commented on HIVE-23210:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
47s{color} | {color:blue} llap-server in master has 89 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} llap-server: The patch generated 2 new + 13 unchanged
- 0 fixed = 15 total (was 13) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21756/dev-support/hive-personality.sh
|
| git revision | master / c3fb689 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21756/yetus/diff-checkstyle-llap-server.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21756/yetus/patch-asflicense-problems.txt
|
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21756/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix shortestjobcomparator when jobs submitted have 1 task their vertices
>
>
> Key: HIVE-23210
> URL: https://issues.apache.org/jira/browse/HIVE-23210
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Panagiotis Garefalakis
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23210.01.patch, HIVE-23210.02.patch,
> HIVE-23210.03.patch, HIVE-23210.wip.patch, TestShortestJobFirstComparator.java
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In latency sensitive queries, lots of jobs can have vertices with 1 task.
> Currently shortestjobcomparator does not work correctly and returns tasks in
> random order.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51]
> This causes delay in the job runtime. I will attach a simple test case
> shortly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[
https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086179#comment-17086179
]
Hive QA commented on HIVE-23210:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13000268/HIVE-23210.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17118 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=87)
[intersect_all.q,unionDistinct_1.q,orc_ppd_schema_evol_3a.q,import_exported_table.q,tez_union_dynamic_partition.q,vector_offset_limit.q,orc_merge10.q,tez_union_dynamic_partition_2.q,temp_table_external.q,llap_udf.q,schemeAuthority.q,cte_2.q,rcfile_createas1.q,cttl.q,remote_script.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21733/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21733/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21733/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13000268 - PreCommit-HIVE-Build
> Fix shortestjobcomparator when jobs submitted have 1 task their vertices
>
>
> Key: HIVE-23210
> URL: https://issues.apache.org/jira/browse/HIVE-23210
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Panagiotis Garefalakis
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23210.01.patch, HIVE-23210.02.patch,
> HIVE-23210.wip.patch, TestShortestJobFirstComparator.java
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In latency sensitive queries, lots of jobs can have vertices with 1 task.
> Currently shortestjobcomparator does not work correctly and returns tasks in
> random order.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51]
> This causes delay in the job runtime. I will attach a simple test case
> shortly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[
https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086153#comment-17086153
]
Hive QA commented on HIVE-23210:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} llap-server in master has 89 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} llap-server: The patch generated 2 new + 13 unchanged
- 0 fixed = 15 total (was 13) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21733/dev-support/hive-personality.sh
|
| git revision | master / 51dda9a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21733/yetus/diff-checkstyle-llap-server.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21733/yetus/patch-asflicense-problems.txt
|
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21733/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix shortestjobcomparator when jobs submitted have 1 task their vertices
>
>
> Key: HIVE-23210
> URL: https://issues.apache.org/jira/browse/HIVE-23210
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Panagiotis Garefalakis
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23210.01.patch, HIVE-23210.02.patch,
> HIVE-23210.wip.patch, TestShortestJobFirstComparator.java
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In latency sensitive queries, lots of jobs can have vertices with 1 task.
> Currently shortestjobcomparator does not work correctly and returns tasks in
> random order.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51]
> This causes delay in the job runtime. I will attach a simple test case
> shortly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[ https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085689#comment-17085689 ] Panagiotis Garefalakis commented on HIVE-23210: --- [~rajesh.balamohan] thanks for clearing this out! The equality comparison is now done in the final return statement: [https://github.com/apache/hive/pull/984/files#diff-f886866e2a28c028d82e35d2a4ebc288R77] The appropriate test case is added here: [https://github.com/apache/hive/pull/984/files#diff-ae97ef12823d4e9f10f596a95340cb3cR364] IMHO the equality scenario (equal waitTime and pending tasks) should be pretty rate but we could push the comparison further up (as Rajesh did in his patch) to avoid doing the ratio divisions. Thoughts? > Fix shortestjobcomparator when jobs submitted have 1 task their vertices > > > Key: HIVE-23210 > URL: https://issues.apache.org/jira/browse/HIVE-23210 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23210.01.patch, HIVE-23210.02.patch, > HIVE-23210.wip.patch, TestShortestJobFirstComparator.java > > Time Spent: 10m > Remaining Estimate: 0h > > In latency sensitive queries, lots of jobs can have vertices with 1 task. > Currently shortestjobcomparator does not work correctly and returns tasks in > random order. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51] > This causes delay in the job runtime. I will attach a simple test case > shortly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[ https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085658#comment-17085658 ] Rajesh Balamohan commented on HIVE-23210: - Issue is not only related to single task. It can happen whenever waitTime and knownPending tasks are equal and the tie has to be released. It does handle the case when you pointed out as higher waittime (in such case it does not fall into this codepath). > Fix shortestjobcomparator when jobs submitted have 1 task their vertices > > > Key: HIVE-23210 > URL: https://issues.apache.org/jira/browse/HIVE-23210 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23210.01.patch, HIVE-23210.02.patch, > HIVE-23210.wip.patch, TestShortestJobFirstComparator.java > > Time Spent: 10m > Remaining Estimate: 0h > > In latency sensitive queries, lots of jobs can have vertices with 1 task. > Currently shortestjobcomparator does not work correctly and returns tasks in > random order. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51] > This causes delay in the job runtime. I will attach a simple test case > shortly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[ https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085655#comment-17085655 ] Panagiotis Garefalakis commented on HIVE-23210: --- [~rajesh.balamohan] thanks for the sample patch! While it does solve the particular corner case it does not properly handle the scenario where there is a comparison between a single-task DAG (wait-time 0) with a multi-task DAG (1 out of X tasks left with greater wait-time) – In that case the proposed approach will still return 0 while we should promote the DAG that waited longer. I tried to tackle those issues in [^HIVE-23210.02.patch] and added some more test cases – could you please take a look? > Fix shortestjobcomparator when jobs submitted have 1 task their vertices > > > Key: HIVE-23210 > URL: https://issues.apache.org/jira/browse/HIVE-23210 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23210.01.patch, HIVE-23210.02.patch, > HIVE-23210.wip.patch, TestShortestJobFirstComparator.java > > Time Spent: 10m > Remaining Estimate: 0h > > In latency sensitive queries, lots of jobs can have vertices with 1 task. > Currently shortestjobcomparator does not work correctly and returns tasks in > random order. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51] > This causes delay in the job runtime. I will attach a simple test case > shortly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[ https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085647#comment-17085647 ] Rajesh Balamohan commented on HIVE-23210: - [^HIVE-23210.wip.patch] : Sample patch that takes care of the corner cases. In this case, if "waitTimes" and "pendingTasks" are same, it compares current attempt's starttime to resolve the conflict. > Fix shortestjobcomparator when jobs submitted have 1 task their vertices > > > Key: HIVE-23210 > URL: https://issues.apache.org/jira/browse/HIVE-23210 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23210.01.patch, HIVE-23210.wip.patch, > TestShortestJobFirstComparator.java > > Time Spent: 10m > Remaining Estimate: 0h > > In latency sensitive queries, lots of jobs can have vertices with 1 task. > Currently shortestjobcomparator does not work correctly and returns tasks in > random order. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51] > This causes delay in the job runtime. I will attach a simple test case > shortly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[
https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085381#comment-17085381
]
Hive QA commented on HIVE-23210:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13000196/HIVE-23210.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17131 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorParallelism
(batchId=294)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21709/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21709/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21709/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13000196 - PreCommit-HIVE-Build
> Fix shortestjobcomparator when jobs submitted have 1 task their vertices
>
>
> Key: HIVE-23210
> URL: https://issues.apache.org/jira/browse/HIVE-23210
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Panagiotis Garefalakis
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23210.01.patch, TestShortestJobFirstComparator.java
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In latency sensitive queries, lots of jobs can have vertices with 1 task.
> Currently shortestjobcomparator does not work correctly and returns tasks in
> random order.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51]
> This causes delay in the job runtime. I will attach a simple test case
> shortly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[
https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085362#comment-17085362
]
Hive QA commented on HIVE-23210:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} llap-server in master has 89 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} llap-server: The patch generated 4 new + 10 unchanged
- 3 fixed = 14 total (was 13) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21709/dev-support/hive-personality.sh
|
| git revision | master / 661e40a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21709/yetus/diff-checkstyle-llap-server.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21709/yetus/patch-asflicense-problems.txt
|
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21709/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix shortestjobcomparator when jobs submitted have 1 task their vertices
>
>
> Key: HIVE-23210
> URL: https://issues.apache.org/jira/browse/HIVE-23210
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Panagiotis Garefalakis
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23210.01.patch, TestShortestJobFirstComparator.java
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In latency sensitive queries, lots of jobs can have vertices with 1 task.
> Currently shortestjobcomparator does not work correctly and returns tasks in
> random order.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51]
> This causes delay in the job runtime. I will attach a simple test case
> shortly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[ https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085219#comment-17085219 ] Panagiotis Garefalakis commented on HIVE-23210: --- [~gopalv] thanks for the comment! You are right, the negative sign can indeed complicate things there. Reworked the patch to handle single-task DAGs as a separate case and added an extra test for multi- vs single-task comparison: [https://github.com/apache/hive/pull/984/files#diff-ae97ef12823d4e9f10f596a95340cb3cR318] Please let me know what you think. > Fix shortestjobcomparator when jobs submitted have 1 task their vertices > > > Key: HIVE-23210 > URL: https://issues.apache.org/jira/browse/HIVE-23210 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23210.01.patch, TestShortestJobFirstComparator.java > > Time Spent: 10m > Remaining Estimate: 0h > > In latency sensitive queries, lots of jobs can have vertices with 1 task. > Currently shortestjobcomparator does not work correctly and returns tasks in > random order. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51] > This causes delay in the job runtime. I will attach a simple test case > shortly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[ https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085184#comment-17085184 ] Gopal Vijayaraghavan commented on HIVE-23210: - This approach might have issues outside of the 1 task setup, because the ratios for one scenario is -ve (i.e 1/1), while the almost equivalent (12/13) gets a +ve number or 0, even though both have only 1 task pending. > Fix shortestjobcomparator when jobs submitted have 1 task their vertices > > > Key: HIVE-23210 > URL: https://issues.apache.org/jira/browse/HIVE-23210 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23210.01.patch, TestShortestJobFirstComparator.java > > Time Spent: 10m > Remaining Estimate: 0h > > In latency sensitive queries, lots of jobs can have vertices with 1 task. > Currently shortestjobcomparator does not work correctly and returns tasks in > random order. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51] > This causes delay in the job runtime. I will attach a simple test case > shortly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[ https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085092#comment-17085092 ] Ashutosh Chauhan commented on HIVE-23210: - [~gopalv] or [~rajesh.balamohan] can you please review Panos's patch? > Fix shortestjobcomparator when jobs submitted have 1 task their vertices > > > Key: HIVE-23210 > URL: https://issues.apache.org/jira/browse/HIVE-23210 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23210.01.patch, TestShortestJobFirstComparator.java > > Time Spent: 10m > Remaining Estimate: 0h > > In latency sensitive queries, lots of jobs can have vertices with 1 task. > Currently shortestjobcomparator does not work correctly and returns tasks in > random order. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51] > This causes delay in the job runtime. I will attach a simple test case > shortly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[
https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084374#comment-17084374
]
Gopal Vijayaraghavan commented on HIVE-23210:
-
{code}
// Compute knownPending tasks. selfAndUpstream indicates task counts for
current vertex and
// it's parent hierarchy. selfAndUpstreamComplete indicates how many of
these have completed.
int knownPending1 = fri1.getNumSelfAndUpstreamTasks() -
fri1.getNumSelfAndUpstreamCompletedTasks();
int knownPending2 = fri2.getNumSelfAndUpstreamTasks() -
fri2.getNumSelfAndUpstreamCompletedTasks();
// longer the wait time for an attempt wrt to its start time, higher the
priority it gets
long waitTime1 = fri1.getCurrentAttemptStartTime() -
fri1.getFirstAttemptStartTime();
long waitTime2 = fri2.getCurrentAttemptStartTime() -
fri2.getFirstAttemptStartTime();
if (waitTime1 == 0 || waitTime2 == 0) {
return knownPending1 - knownPending2;
}
{code}
So here, for a vertex with 1 task, for the first attempt, it will return 0;
It should return at least the FIFO order of getCurrentAttemptStartTime()
> Fix shortestjobcomparator when jobs submitted have 1 task their vertices
>
>
> Key: HIVE-23210
> URL: https://issues.apache.org/jira/browse/HIVE-23210
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Panagiotis Garefalakis
>Priority: Major
> Attachments: TestShortestJobFirstComparator.java
>
>
> In latency sensitive queries, lots of jobs can have vertices with 1 task.
> Currently shortestjobcomparator does not work correctly and returns tasks in
> random order.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51]
> This causes delay in the job runtime. I will attach a simple test case
> shortly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[ https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084232#comment-17084232 ] Ashutosh Chauhan commented on HIVE-23210: - I see. Make sense. > Fix shortestjobcomparator when jobs submitted have 1 task their vertices > > > Key: HIVE-23210 > URL: https://issues.apache.org/jira/browse/HIVE-23210 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Major > Attachments: TestShortestJobFirstComparator.java > > > In latency sensitive queries, lots of jobs can have vertices with 1 task. > Currently shortestjobcomparator does not work correctly and returns tasks in > random order. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51] > This causes delay in the job runtime. I will attach a simple test case > shortly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[ https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084221#comment-17084221 ] Gopal Vijayaraghavan commented on HIVE-23210: - bq. Are you suggesting to run single task jobs at higher priority compared to rest of tasks in daemon. No, this is specifically for the comparator which has 1 task vertices on both sides, not that scenario. The current problem is that the comparator returns 0 when comparing 1 task vertices - which causes them to be scheduled at random rather than in the order of the request time or by any other meaningful priority. > Fix shortestjobcomparator when jobs submitted have 1 task their vertices > > > Key: HIVE-23210 > URL: https://issues.apache.org/jira/browse/HIVE-23210 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Major > Attachments: TestShortestJobFirstComparator.java > > > In latency sensitive queries, lots of jobs can have vertices with 1 task. > Currently shortestjobcomparator does not work correctly and returns tasks in > random order. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51] > This causes delay in the job runtime. I will attach a simple test case > shortly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[ https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084168#comment-17084168 ] Ashutosh Chauhan commented on HIVE-23210: - [~rajesh.balamohan] Do you have proposed solution for it as well? Are you suggesting to run single task jobs at higher priority compared to rest of tasks in daemon. That doesn't sound fair as it may starve multi-task jobs if you have steady stream of these single task jobs. > Fix shortestjobcomparator when jobs submitted have 1 task their vertices > > > Key: HIVE-23210 > URL: https://issues.apache.org/jira/browse/HIVE-23210 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Major > Attachments: TestShortestJobFirstComparator.java > > > In latency sensitive queries, lots of jobs can have vertices with 1 task. > Currently shortestjobcomparator does not work correctly and returns tasks in > random order. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51] > This causes delay in the job runtime. I will attach a simple test case > shortly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23210) Fix shortestjobcomparator when jobs submitted have 1 task their vertices
[ https://issues.apache.org/jira/browse/HIVE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083880#comment-17083880 ] Rajesh Balamohan commented on HIVE-23210: - TestShortestJobFirstComparator::testWaitQueueAging should reproduce the issue. In some cases, tasks which got submitted initially gets picked up at the last. > Fix shortestjobcomparator when jobs submitted have 1 task their vertices > > > Key: HIVE-23210 > URL: https://issues.apache.org/jira/browse/HIVE-23210 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Major > Attachments: TestShortestJobFirstComparator.java > > > In latency sensitive queries, lots of jobs can have vertices with 1 task. > Currently shortestjobcomparator does not work correctly and returns tasks in > random order. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator/ShortestJobFirstComparator.java#L51] > This causes delay in the job runtime. I will attach a simple test case > shortly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
