[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077181#comment-17077181 ] László Bodor commented on HIVE-23122: - pushed to master! thanks [~rajesh.balamohan] and [~pgaref] for the review > LLAP: TaskExecutorService should log details about task eviction decision > details > - > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23122.02.patch, HIVE-23122.03.patch, > HIVE-23122.03.patch > > > TaskExecutorService maintains a waitQueue, and can evict a task in favor of > another. Under the hood, the queue uses a configurable > [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator]. > The currently available comparators typically use the following properties > of a task(wrapper): > getWithinDagPriority: related to vertex > currentAttemptStartTime > firstAttemptStartTime > knownPending: remaining upstream tasks > The problem is, when an eviction happens, the INFO level message doesn't > provide any insight about the decision, only attempts ids like below: > {code} > attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of > attempt_1585248378306_0003_175_02_79_175 because of lower priority > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077173#comment-17077173 ] Hive QA commented on HIVE-23122: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12999080/HIVE-23122.03.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 18194 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/21487/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21487/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21487/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12999080 - PreCommit-HIVE-Build > LLAP: TaskExecutorService should log details about task eviction decision > details > - > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23122.02.patch, HIVE-23122.03.patch, > HIVE-23122.03.patch > > > TaskExecutorService maintains a waitQueue, and can evict a task in favor of > another. Under the hood, the queue uses a configurable > [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator]. > The currently available comparators typically use the following properties > of a task(wrapper): > getWithinDagPriority: related to vertex > currentAttemptStartTime > firstAttemptStartTime > knownPending: remaining upstream tasks > The problem is, when an eviction happens, the INFO level message doesn't > provide any insight about the decision, only attempts ids like below: > {code} > attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of > attempt_1585248378306_0003_175_02_79_175 because of lower priority > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077143#comment-17077143 ] Hive QA commented on HIVE-23122: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 5s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 41s{color} | {color:blue} llap-server in master has 90 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 13m 19s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-21487/dev-support/hive-personality.sh | | git revision | master / 897e1e8 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | modules | C: llap-server U: llap-server | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-21487/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > LLAP: TaskExecutorService should log details about task eviction decision > details > - > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23122.02.patch, HIVE-23122.03.patch, > HIVE-23122.03.patch > > > TaskExecutorService maintains a waitQueue, and can evict a task in favor of > another. Under the hood, the queue uses a configurable > [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator]. > The currently available comparators typically use the following properties > of a task(wrapper): > getWithinDagPriority: related to vertex > currentAttemptStartTime > firstAttemptStartTime > knownPending: remaining upstream tasks > The problem is, when an eviction happens, the INFO level message doesn't > provide any insight about the decision, only attempts ids like below: > {code} > attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of > attempt_1585248378306_0003_175_02_79_175 because of lower priority > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17074939#comment-17074939 ] Hive QA commented on HIVE-23122: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12998710/HIVE-23122.03.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 18172 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.metastore.TestMetastoreHousekeepingLeaderEmptyConfig.testHouseKeepingThreadExistence (batchId=252) org.apache.hive.service.TestDFSErrorHandling.org.apache.hive.service.TestDFSErrorHandling (batchId=290) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/21427/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21427/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21427/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12998710 - PreCommit-HIVE-Build > LLAP: TaskExecutorService should log details about task eviction decision > details > - > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23122.02.patch, HIVE-23122.03.patch > > > TaskExecutorService maintains a waitQueue, and can evict a task in favor of > another. Under the hood, the queue uses a configurable > [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator]. > The currently available comparators typically use the following properties > of a task(wrapper): > getWithinDagPriority: related to vertex > currentAttemptStartTime > firstAttemptStartTime > knownPending: remaining upstream tasks > The problem is, when an eviction happens, the INFO level message doesn't > provide any insight about the decision, only attempts ids like below: > {code} > attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of > attempt_1585248378306_0003_175_02_79_175 because of lower priority > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17074914#comment-17074914 ] Hive QA commented on HIVE-23122: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 29s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 43s{color} | {color:blue} llap-server in master has 90 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 13m 47s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-21427/dev-support/hive-personality.sh | | git revision | master / dea35b4 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | modules | C: llap-server U: llap-server | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-21427/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > LLAP: TaskExecutorService should log details about task eviction decision > details > - > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23122.02.patch, HIVE-23122.03.patch > > > TaskExecutorService maintains a waitQueue, and can evict a task in favor of > another. Under the hood, the queue uses a configurable > [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator]. > The currently available comparators typically use the following properties > of a task(wrapper): > getWithinDagPriority: related to vertex > currentAttemptStartTime > firstAttemptStartTime > knownPending: remaining upstream tasks > The problem is, when an eviction happens, the INFO level message doesn't > provide any insight about the decision, only attempts ids like below: > {code} > attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of > attempt_1585248378306_0003_175_02_79_175 because of lower priority > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073834#comment-17073834 ] Hive QA commented on HIVE-23122: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12998463/HIVE-23122.02.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 18162 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/21388/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21388/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21388/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12998463 - PreCommit-HIVE-Build > LLAP: TaskExecutorService should log details about task eviction decision > details > - > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23122.02.patch > > > TaskExecutorService maintains a waitQueue, and can evict a task in favor of > another. Under the hood, the queue uses a configurable > [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator]. > The currently available comparators typically use the following properties > of a task(wrapper): > getWithinDagPriority: related to vertex > currentAttemptStartTime > firstAttemptStartTime > knownPending: remaining upstream tasks > The problem is, when an eviction happens, the INFO level message doesn't > provide any insight about the decision, only attempts ids like below: > {code} > attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of > attempt_1585248378306_0003_175_02_79_175 because of lower priority > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073784#comment-17073784 ] Hive QA commented on HIVE-23122: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 50s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 44s{color} | {color:blue} llap-server in master has 90 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 14m 14s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-21388/dev-support/hive-personality.sh | | git revision | master / 8739c58 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | modules | C: llap-server U: llap-server | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-21388/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > LLAP: TaskExecutorService should log details about task eviction decision > details > - > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23122.02.patch > > > TaskExecutorService maintains a waitQueue, and can evict a task in favor of > another. Under the hood, the queue uses a configurable > [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator]. > The currently available comparators typically use the following properties > of a task(wrapper): > getWithinDagPriority: related to vertex > currentAttemptStartTime > firstAttemptStartTime > knownPending: remaining upstream tasks > The problem is, when an eviction happens, the INFO level message doesn't > provide any insight about the decision, only attempts ids like below: > {code} > attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of > attempt_1585248378306_0003_175_02_79_175 because of lower priority > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073727#comment-17073727 ] László Bodor commented on HIVE-23122: - thanks for taking a look [~rajesh.balamohan]! are you sure that it's needed on DEBUG level? it's already on INFO level, and this logging only involves primitives, so an expectable amount of characters...perhaps we can add the new extended line on DEBUG level, and the old one on INFO level? > LLAP: TaskExecutorService should log details about task eviction decision > details > - > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23122.02.patch > > > TaskExecutorService maintains a waitQueue, and can evict a task in favor of > another. Under the hood, the queue uses a configurable > [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator]. > The currently available comparators typically use the following properties > of a task(wrapper): > getWithinDagPriority: related to vertex > currentAttemptStartTime > firstAttemptStartTime > knownPending: remaining upstream tasks > The problem is, when an eviction happens, the INFO level message doesn't > provide any insight about the decision, only attempts ids like below: > {code} > attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of > attempt_1585248378306_0003_175_02_79_175 because of lower priority > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073717#comment-17073717 ] Rajesh Balamohan commented on HIVE-23122: - Given that this is in hotpath, we can move to debug level, if this adds significant logging in certain jobs. +1. > LLAP: TaskExecutorService should log details about task eviction decision > details > - > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23122.02.patch > > > TaskExecutorService maintains a waitQueue, and can evict a task in favor of > another. Under the hood, the queue uses a configurable > [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator]. > The currently available comparators typically use the following properties > of a task(wrapper): > getWithinDagPriority: related to vertex > currentAttemptStartTime > firstAttemptStartTime > knownPending: remaining upstream tasks > The problem is, when an eviction happens, the INFO level message doesn't > provide any insight about the decision, only attempts ids like below: > {code} > attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of > attempt_1585248378306_0003_175_02_79_175 because of lower priority > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072821#comment-17072821 ] László Bodor commented on HIVE-23122: - cc: [~rajesh.balamohan] > LLAP: TaskExecutorService should log details about task eviction decision > details > - > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23122.02.patch > > > TaskExecutorService maintains a waitQueue, and can evict a task in favor of > another. Under the hood, the queue uses a configurable > [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator]. > The currently available comparators typically use the following properties > of a task(wrapper): > getWithinDagPriority: related to vertex > currentAttemptStartTime > firstAttemptStartTime > knownPending: remaining upstream tasks > The problem is, when an eviction happens, the INFO level message doesn't > provide any insight about the decision, only attempts ids like below: > {code} > attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of > attempt_1585248378306_0003_175_02_79_175 because of lower priority > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072785#comment-17072785 ] Panagiotis Garefalakis commented on HIVE-23122: --- Thanks [~abstractdog] +1 tests pending > LLAP: TaskExecutorService should log details about task eviction decision > details > - > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23122.02.patch > > > TaskExecutorService maintains a waitQueue, and can evict a task in favor of > another. Under the hood, the queue uses a configurable > [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator]. > The currently available comparators typically use the following properties > of a task(wrapper): > getWithinDagPriority: related to vertex > currentAttemptStartTime > firstAttemptStartTime > knownPending: remaining upstream tasks > The problem is, when an eviction happens, the INFO level message doesn't > provide any insight about the decision, only attempts ids like below: > {code} > attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of > attempt_1585248378306_0003_175_02_79_175 because of lower priority > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072783#comment-17072783 ] László Bodor commented on HIVE-23122: - thanks for the comments [~pgaref], included canFinishForPriority removing 01.patch for saving ptest time > LLAP TaskExecutorService should log details about task eviction decision > details > > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23122.02.patch > > > TaskExecutorService maintains a waitQueue, and can evict a task in favor of > another. Under the hood, the queue uses a configurable > [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator]. > The currently available comparators typically use the following properties > of a task(wrapper): > getWithinDagPriority: related to vertex > currentAttemptStartTime > firstAttemptStartTime > knownPending: remaining upstream tasks > The problem is, when an eviction happens, the INFO level message doesn't > provide any insight about the decision, only attempts ids like below: > {code} > attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of > attempt_1585248378306_0003_175_02_79_175 because of lower priority > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072716#comment-17072716 ] Panagiotis Garefalakis commented on HIVE-23122: --- Thanks for the patch [~abstractdog] – pretty useful – would it make sense to add the canFinishForPriority property used by the BaseComparator as well? > LLAP TaskExecutorService should log details about task eviction decision > details > > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23122.01.patch > > > TaskExecutorService maintains a waitQueue, and can evict a task in favor of > another. Under the hood, the queue uses a configurable > [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator]. > The currently available comparators typically use the following properties > of a task(wrapper): > getWithinDagPriority: related to vertex > currentAttemptStartTime > firstAttemptStartTime > knownPending: remaining upstream tasks > The problem is, when an eviction happens, the INFO level message doesn't > provide any insight about the decision, only attempts ids like below: > {code} > attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of > attempt_1585248378306_0003_175_02_79_175 because of lower priority > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)