[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077181#comment-17077181
]
László Bodor commented on HIVE-23122:
-
pushed to master! thanks [~rajesh.balamohan] and [~pgaref] for the review
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch, HIVE-23122.03.patch,
> HIVE-23122.03.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077173#comment-17077173
]
Hive QA commented on HIVE-23122:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12999080/HIVE-23122.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18194 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21487/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21487/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21487/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12999080 - PreCommit-HIVE-Build
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch, HIVE-23122.03.patch,
> HIVE-23122.03.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077143#comment-17077143
]
Hive QA commented on HIVE-23122:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21487/dev-support/hive-personality.sh
|
| git revision | master / 897e1e8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21487/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch, HIVE-23122.03.patch,
> HIVE-23122.03.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074939#comment-17074939
]
Hive QA commented on HIVE-23122:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998710/HIVE-23122.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 18172 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestMetastoreHousekeepingLeaderEmptyConfig.testHouseKeepingThreadExistence
(batchId=252)
org.apache.hive.service.TestDFSErrorHandling.org.apache.hive.service.TestDFSErrorHandling
(batchId=290)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21427/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21427/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21427/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998710 - PreCommit-HIVE-Build
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch, HIVE-23122.03.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074914#comment-17074914
]
Hive QA commented on HIVE-23122:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21427/dev-support/hive-personality.sh
|
| git revision | master / dea35b4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21427/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch, HIVE-23122.03.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073834#comment-17073834
]
Hive QA commented on HIVE-23122:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998463/HIVE-23122.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18162 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21388/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21388/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21388/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998463 - PreCommit-HIVE-Build
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073784#comment-17073784
]
Hive QA commented on HIVE-23122:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21388/dev-support/hive-personality.sh
|
| git revision | master / 8739c58 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21388/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073727#comment-17073727
]
László Bodor commented on HIVE-23122:
-
thanks for taking a look [~rajesh.balamohan]! are you sure that it's needed on
DEBUG level? it's already on INFO level, and this logging only involves
primitives, so an expectable amount of characters...perhaps we can add the new
extended line on DEBUG level, and the old one on INFO level?
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073717#comment-17073717
]
Rajesh Balamohan commented on HIVE-23122:
-
Given that this is in hotpath, we can move to debug level, if this adds
significant logging in certain jobs.
+1.
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072821#comment-17072821
]
László Bodor commented on HIVE-23122:
-
cc: [~rajesh.balamohan]
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072785#comment-17072785
]
Panagiotis Garefalakis commented on HIVE-23122:
---
Thanks [~abstractdog]
+1 tests pending
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072783#comment-17072783
]
László Bodor commented on HIVE-23122:
-
thanks for the comments [~pgaref], included canFinishForPriority
removing 01.patch for saving ptest time
> LLAP TaskExecutorService should log details about task eviction decision
> details
>
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072716#comment-17072716
]
Panagiotis Garefalakis commented on HIVE-23122:
---
Thanks for the patch [~abstractdog] – pretty useful – would it make sense to
add the canFinishForPriority property used by the BaseComparator as well?
> LLAP TaskExecutorService should log details about task eviction decision
> details
>
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.01.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
