[jira] [Commented] (HIVE-20772) record per-task CPU counters in LLAP

2018-10-18 Thread Sergey Shelukhin (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655821#comment-16655821
 ] 

Sergey Shelukhin commented on HIVE-20772:
-

[~prasanth_j] can you take a look?

> record per-task CPU counters in LLAP
> 
>
> Key: HIVE-20772
> URL: https://issues.apache.org/jira/browse/HIVE-20772
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20772.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20772) record per-task CPU counters in LLAP

2018-10-18 Thread Sergey Shelukhin (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655820#comment-16655820
 ] 

Sergey Shelukhin commented on HIVE-20772:
-

A quick patch... I'd probably need to run it in a cluster to make sure it works 
correctly in a realistic deployment.


> record per-task CPU counters in LLAP
> 
>
> Key: HIVE-20772
> URL: https://issues.apache.org/jira/browse/HIVE-20772
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20772.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20772) record per-task CPU counters in LLAP

2018-10-18 Thread Prasanth Jayachandran (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655829#comment-16655829
 ] 

Prasanth Jayachandran commented on HIVE-20772:
--

tezCounters.findCounter(LlapExecutorCounters.EXECUTOR_CPU_NS).increment(cpuTime);

Should this be set instead of increment? If we are reusing the executor thread 
(I hope we are not), increment will give aggregate whereas set will just give 
just for that thread. 

 

Looks good otherwise. 

> record per-task CPU counters in LLAP
> 
>
> Key: HIVE-20772
> URL: https://issues.apache.org/jira/browse/HIVE-20772
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20772.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20772) record per-task CPU counters in LLAP

2018-10-18 Thread Sergey Shelukhin (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655832#comment-16655832
 ] 

Sergey Shelukhin commented on HIVE-20772:
-

Hmm, I was assuming based on other counters there (FS ones) that this object is 
scoped per task... e.g. {noformat}   tezCounters = 
taskRunner2Callable.addAndGetTezCounter(FileSystemCounter.class.getName());
  } else if (actualCallable instanceof TezCounterSource) { {noformat}


> record per-task CPU counters in LLAP
> 
>
> Key: HIVE-20772
> URL: https://issues.apache.org/jira/browse/HIVE-20772
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20772.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20772) record per-task CPU counters in LLAP

2018-10-18 Thread Prasanth Jayachandran (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655838#comment-16655838
 ] 

Prasanth Jayachandran commented on HIVE-20772:
--

oh yeah.. make sense :)

+1

> record per-task CPU counters in LLAP
> 
>
> Key: HIVE-20772
> URL: https://issues.apache.org/jira/browse/HIVE-20772
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20772.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20772) record per-task CPU counters in LLAP

2018-10-19 Thread Hive QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657023#comment-16657023
 ] 

Hive QA commented on HIVE-20772:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
29s{color} | {color:blue} llap-common in master has 71 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
40s{color} | {color:blue} llap-server in master has 83 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
20s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
18s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 18s{color} 
| {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
13s{color} | {color:red} llap-server: The patch generated 3 new + 223 unchanged 
- 1 fixed = 226 total (was 224) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
18s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 41s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14569/dev-support/hive-personality.sh
 |
| git revision | master / bd3c05d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14569/yetus/patch-mvninstall-llap-server.txt
 |
| compile | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14569/yetus/patch-compile-llap-server.txt
 |
| javac | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14569/yetus/patch-compile-llap-server.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14569/yetus/diff-checkstyle-llap-server.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14569/yetus/patch-findbugs-llap-server.txt
 |
| modules | C: llap-common llap-server U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14569/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> record per-task CPU counters in LLAP
> 
>
> Key: HIVE-20772
> URL: https://issues.apache.org/jira/browse/HIVE-20772
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20772.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20772) record per-task CPU counters in LLAP

2018-10-19 Thread Hive QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657108#comment-16657108
 ] 

Hive QA commented on HIVE-20772:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944615/HIVE-20772.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 15111 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters1]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic] 
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
 (batchId=155)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/14569/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14569/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14569/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12944615 - PreCommit-HIVE-Build

> record per-task CPU counters in LLAP
> 
>
> Key: HIVE-20772
> URL: https://issues.apache.org/jira/browse/HIVE-20772
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20772.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20772) record per-task CPU counters in LLAP

2018-10-19 Thread Sergey Shelukhin (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657446#comment-16657446
 ] 

Sergey Shelukhin commented on HIVE-20772:
-

Fixed an issue where the wrapper callable from the IO is not being handled 
correctly by Stats thread pool.

> record per-task CPU counters in LLAP
> 
>
> Key: HIVE-20772
> URL: https://issues.apache.org/jira/browse/HIVE-20772
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20772.01.patch, HIVE-20772.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20772) record per-task CPU counters in LLAP

2018-10-20 Thread Hive QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657848#comment-16657848
 ] 

Hive QA commented on HIVE-20772:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
34s{color} | {color:blue} llap-common in master has 71 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
45s{color} | {color:blue} llap-server in master has 83 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
21s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
22s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 22s{color} 
| {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
18s{color} | {color:red} llap-server: The patch generated 6 new + 223 unchanged 
- 1 fixed = 229 total (was 224) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
23s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 17s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14590/dev-support/hive-personality.sh
 |
| git revision | master / 7ed8bf4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14590/yetus/patch-mvninstall-llap-server.txt
 |
| compile | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14590/yetus/patch-compile-llap-server.txt
 |
| javac | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14590/yetus/patch-compile-llap-server.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14590/yetus/diff-checkstyle-llap-server.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14590/yetus/patch-findbugs-llap-server.txt
 |
| modules | C: llap-common llap-server U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14590/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> record per-task CPU counters in LLAP
> 
>
> Key: HIVE-20772
> URL: https://issues.apache.org/jira/browse/HIVE-20772
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20772.01.patch, HIVE-20772.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20772) record per-task CPU counters in LLAP

2018-10-20 Thread Hive QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657860#comment-16657860
 ] 

Hive QA commented on HIVE-20772:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944807/HIVE-20772.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15109 tests 
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=196)

[druidmini_masking.q,druidmini_test1.q,druidkafkamini_basic.q,druidmini_joins.q,druid_timestamptz.q]
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/14590/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14590/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14590/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12944807 - PreCommit-HIVE-Build

> record per-task CPU counters in LLAP
> 
>
> Key: HIVE-20772
> URL: https://issues.apache.org/jira/browse/HIVE-20772
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20772.01.patch, HIVE-20772.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20772) record per-task CPU counters in LLAP

2018-10-23 Thread Sergey Shelukhin (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661379#comment-16661379
 ] 

Sergey Shelukhin commented on HIVE-20772:
-

{noformat}
+ java -cp 
'/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/hive-ptest-3.0-classes.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/lib/*'
 org.apache.hive.ptest.api.client.PTestClient --command testStart --outputDir 
/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target
 --password '[***]' --testHandle PreCommit-HIVE-Build-14608 --endpoint 
http://104.198.109.242:8080/hive-ptest-3.0 --logsEndpoint 
http://104.198.109.242/logs/ --profile master-mr2 --patch 
https://issues.apache.org/jira/secure/attachment/12945103/HIVE-20772.02.patch 
--jira HIVE-20772
Build timed out (after 500 minutes). Marking the build as aborted.
{noformat}
no test reports... not sure what's up with ptest

> record per-task CPU counters in LLAP
> 
>
> Key: HIVE-20772
> URL: https://issues.apache.org/jira/browse/HIVE-20772
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20772.01.patch, HIVE-20772.02.patch, 
> HIVE-20772.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)