[jira] [Commented] (HIVE-19810) StorageHandler fail to ship jars in Tez intermittently

2018-06-15 Thread Sergey Shelukhin (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-19810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514227#comment-16514227
 ] 

Sergey Shelukhin commented on HIVE-19810:
-

+1. Did you test the patch on some cluster, since there's no unit test? 

> StorageHandler fail to ship jars in Tez intermittently
> --
>
> Key: HIVE-19810
> URL: https://issues.apache.org/jira/browse/HIVE-19810
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19810.1.patch, HIVE-19810.2.patch, testcase.patch
>
>
> Hive relies on StorageHandler to ship jars to backend automatically in 
> several cases: JdbcStorageHandler, HBaseStorageHandler, 
> AccumuloStorageHandler. This does not work reliably, in particular, the first 
> dag in the session will have those jars, the second will not unless container 
> is reused. In the later case, the containers allocated to first dag will be 
> reused in the second dag so the container will have additional resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19810) StorageHandler fail to ship jars in Tez intermittently

2018-06-14 Thread Daniel Dai (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-19810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513137#comment-16513137
 ] 

Daniel Dai commented on HIVE-19810:
---

[~sershe], are you able to review this patch?

> StorageHandler fail to ship jars in Tez intermittently
> --
>
> Key: HIVE-19810
> URL: https://issues.apache.org/jira/browse/HIVE-19810
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19810.1.patch, HIVE-19810.2.patch, testcase.patch
>
>
> Hive relies on StorageHandler to ship jars to backend automatically in 
> several cases: JdbcStorageHandler, HBaseStorageHandler, 
> AccumuloStorageHandler. This does not work reliably, in particular, the first 
> dag in the session will have those jars, the second will not unless container 
> is reused. In the later case, the containers allocated to first dag will be 
> reused in the second dag so the container will have additional resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19810) StorageHandler fail to ship jars in Tez intermittently

2018-06-09 Thread Hive QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-19810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507158#comment-16507158
 ] 

Hive QA commented on HIVE-19810:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12927104/HIVE-19810.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 14512 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/11663/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11663/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11663/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12927104 - PreCommit-HIVE-Build

> StorageHandler fail to ship jars in Tez intermittently
> --
>
> Key: HIVE-19810
> URL: https://issues.apache.org/jira/browse/HIVE-19810
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19810.1.patch, HIVE-19810.2.patch, testcase.patch
>
>
> Hive relies on StorageHandler to ship jars to backend automatically in 
> several cases: JdbcStorageHandler, HBaseStorageHandler, 
> AccumuloStorageHandler. This does not work reliably, in particular, the first 
> dag in the session will have those jars, the second will not unless container 
> is reused. In the later case, the containers allocated to first dag will be 
> reused in the second dag so the container will have additional resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19810) StorageHandler fail to ship jars in Tez intermittently

2018-06-09 Thread Hive QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-19810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507143#comment-16507143
 ] 

Hive QA commented on HIVE-19810:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
34s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
20s{color} | {color:blue} ql in master has 2284 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
33s{color} | {color:red} ql: The patch generated 3 new + 24 unchanged - 1 fixed 
= 27 total (was 25) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
11s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 13s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11663/dev-support/hive-personality.sh
 |
| git revision | master / 812ba32 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11663/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11663/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> StorageHandler fail to ship jars in Tez intermittently
> --
>
> Key: HIVE-19810
> URL: https://issues.apache.org/jira/browse/HIVE-19810
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19810.1.patch, HIVE-19810.2.patch, testcase.patch
>
>
> Hive relies on StorageHandler to ship jars to backend automatically in 
> several cases: JdbcStorageHandler, HBaseStorageHandler, 
> AccumuloStorageHandler. This does not work reliably, in particular, the first 
> dag in the session will have those jars, the second will not unless container 
> is reused. In the later case, the containers allocated to first dag will be 
> reused in the second dag so the container will have additional resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19810) StorageHandler fail to ship jars in Tez intermittently

2018-06-08 Thread Daniel Dai (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-19810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506477#comment-16506477
 ] 

Daniel Dai commented on HIVE-19810:
---

ptest is testing wrong patch. Retrigger.

> StorageHandler fail to ship jars in Tez intermittently
> --
>
> Key: HIVE-19810
> URL: https://issues.apache.org/jira/browse/HIVE-19810
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19810.1.patch, HIVE-19810.2.patch, testcase.patch
>
>
> Hive relies on StorageHandler to ship jars to backend automatically in 
> several cases: JdbcStorageHandler, HBaseStorageHandler, 
> AccumuloStorageHandler. This does not work reliably, in particular, the first 
> dag in the session will have those jars, the second will not unless container 
> is reused. In the later case, the containers allocated to first dag will be 
> reused in the second dag so the container will have additional resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19810) StorageHandler fail to ship jars in Tez intermittently

2018-06-08 Thread Hive QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-19810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505829#comment-16505829
 ] 

Hive QA commented on HIVE-19810:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926658/testcase.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14509 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
 (batchId=237)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/11612/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11612/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11612/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12926658 - PreCommit-HIVE-Build

> StorageHandler fail to ship jars in Tez intermittently
> --
>
> Key: HIVE-19810
> URL: https://issues.apache.org/jira/browse/HIVE-19810
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19810.1.patch, testcase.patch
>
>
> Hive relies on StorageHandler to ship jars to backend automatically in 
> several cases: JdbcStorageHandler, HBaseStorageHandler, 
> AccumuloStorageHandler. This does not work reliably, in particular, the first 
> dag in the session will have those jars, the second will not unless container 
> is reused. In the later case, the containers allocated to first dag will be 
> reused in the second dag so the container will have additional resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19810) StorageHandler fail to ship jars in Tez intermittently

2018-06-08 Thread Hive QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-19810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505800#comment-16505800
 ] 

Hive QA commented on HIVE-19810:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
33s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
27s{color} | {color:blue} ql in master has 2284 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
33s{color} | {color:red} ql: The patch generated 2 new + 0 unchanged - 0 fixed 
= 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 25s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11612/dev-support/hive-personality.sh
 |
| git revision | master / 13960aa |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11612/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-11612/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> StorageHandler fail to ship jars in Tez intermittently
> --
>
> Key: HIVE-19810
> URL: https://issues.apache.org/jira/browse/HIVE-19810
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19810.1.patch, testcase.patch
>
>
> Hive relies on StorageHandler to ship jars to backend automatically in 
> several cases: JdbcStorageHandler, HBaseStorageHandler, 
> AccumuloStorageHandler. This does not work reliably, in particular, the first 
> dag in the session will have those jars, the second will not unless container 
> is reused. In the later case, the containers allocated to first dag will be 
> reused in the second dag so the container will have additional resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19810) StorageHandler fail to ship jars in Tez intermittently

2018-06-05 Thread Daniel Dai (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-19810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502619#comment-16502619
 ] 

Daniel Dai commented on HIVE-19810:
---

Unit test is tricky, seems tez container reuse properties are not controllable 
in the client side. I don't want to touch MiniTezCluster setting to affect mass 
tests. The attached testcase fails by chance in MiniTezCluster without the 
patch. Similar test on single node cluster failed reliably for me.

> StorageHandler fail to ship jars in Tez intermittently
> --
>
> Key: HIVE-19810
> URL: https://issues.apache.org/jira/browse/HIVE-19810
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19810.1.patch, testcase.patch
>
>
> Hive relies on StorageHandler to ship jars to backend automatically in 
> several cases: JdbcStorageHandler, HBaseStorageHandler, 
> AccumuloStorageHandler. This does not work reliably, in particular, the first 
> dag in the session will have those jars, the second will not unless container 
> is reused. In the later case, the containers allocated to first dag will be 
> reused in the second dag so the container will have additional resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)