[jira] [Commented] (HIVE-11923) allow qtests to run via a single client session for tez and llap
[ https://issues.apache.org/jira/browse/HIVE-11923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14903790#comment-14903790 ] Siddharth Seth commented on HIVE-11923: --- [~sershe] - not much point submitting this to jenkins yet. There's a lot of failures. Some potential issues like 11924, others are caused by the fact that the TransactionManager can be set once per session. Will upload another patch which resets the SessionState, but uses the same AM. > allow qtests to run via a single client session for tez and llap > > > Key: HIVE-11923 > URL: https://issues.apache.org/jira/browse/HIVE-11923 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-11923.1.txt, HIVE-11923.branch-1.txt > > > Launching a new session - AM and containers for each test adds unnecessary > overheads. Running via a single session should reduce the run time > significantly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11923) allow qtests to run via a single client session for tez and llap
[ https://issues.apache.org/jira/browse/HIVE-11923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934343#comment-14934343 ] Prasanth Jayachandran commented on HIVE-11923: -- LGTM, +1. Pending tests and the resolving of the blocking issue. > allow qtests to run via a single client session for tez and llap > > > Key: HIVE-11923 > URL: https://issues.apache.org/jira/browse/HIVE-11923 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-11923.1.txt, HIVE-11923.2.branchllap.txt, > HIVE-11923.2.txt, HIVE-11923.branch-1.txt > > > Launching a new session - AM and containers for each test adds unnecessary > overheads. Running via a single session should reduce the run time > significantly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11923) allow qtests to run via a single client session for tez and llap
[ https://issues.apache.org/jira/browse/HIVE-11923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934411#comment-14934411 ] Prasanth Jayachandran commented on HIVE-11923: -- Committed llap patch to branch. > allow qtests to run via a single client session for tez and llap > > > Key: HIVE-11923 > URL: https://issues.apache.org/jira/browse/HIVE-11923 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-11923.1.txt, HIVE-11923.2.branchllap.txt, > HIVE-11923.2.txt, HIVE-11923.branch-1.txt > > > Launching a new session - AM and containers for each test adds unnecessary > overheads. Running via a single session should reduce the run time > significantly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11923) allow qtests to run via a single client session for tez and llap
[ https://issues.apache.org/jira/browse/HIVE-11923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934445#comment-14934445 ] Siddharth Seth commented on HIVE-11923: --- Removing HIVE-11924 as a blocker. SessionState is being reset in the latest patch - so the issue does not show up while running tests. However, it still appears to be a valid issue. > allow qtests to run via a single client session for tez and llap > > > Key: HIVE-11923 > URL: https://issues.apache.org/jira/browse/HIVE-11923 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-11923.1.txt, HIVE-11923.2.branchllap.txt, > HIVE-11923.2.txt, HIVE-11923.branch-1.txt > > > Launching a new session - AM and containers for each test adds unnecessary > overheads. Running via a single session should reduce the run time > significantly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11923) allow qtests to run via a single client session for tez and llap
[
https://issues.apache.org/jira/browse/HIVE-11923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14942451#comment-14942451
]
Hive QA commented on HIVE-11923:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12764701/HIVE-11923.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9640 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_scriptfile1
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5516/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5516/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5516/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12764701 - PreCommit-HIVE-TRUNK-Build
> allow qtests to run via a single client session for tez and llap
>
>
> Key: HIVE-11923
> URL: https://issues.apache.org/jira/browse/HIVE-11923
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-11923.1.txt, HIVE-11923.2.branchllap.txt,
> HIVE-11923.2.patch, HIVE-11923.2.txt, HIVE-11923.2.txt,
> HIVE-11923.branch-1.txt
>
>
> Launching a new session - AM and containers for each test adds unnecessary
> overheads. Running via a single session should reduce the run time
> significantly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11923) allow qtests to run via a single client session for tez and llap
[ https://issues.apache.org/jira/browse/HIVE-11923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14942461#comment-14942461 ] Prasanth Jayachandran commented on HIVE-11923: -- [~sseth] the test failure looks related. Can you take a look? > allow qtests to run via a single client session for tez and llap > > > Key: HIVE-11923 > URL: https://issues.apache.org/jira/browse/HIVE-11923 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 1.3.0, 2.0.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-11923.1.txt, HIVE-11923.2.branchllap.txt, > HIVE-11923.2.patch, HIVE-11923.2.txt, HIVE-11923.2.txt, > HIVE-11923.branch-1.txt > > > Launching a new session - AM and containers for each test adds unnecessary > overheads. Running via a single session should reduce the run time > significantly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11923) allow qtests to run via a single client session for tez and llap
[ https://issues.apache.org/jira/browse/HIVE-11923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949364#comment-14949364 ] Sergey Shelukhin commented on HIVE-11923: - The failure is HIVE-12072. Before a real fix is available (which as far as I understood might also need Tez changes), can you exclude this test from reuse somehow? Perhaps via a new magic test variable akin to SORT_QUERY_RESULTS? > allow qtests to run via a single client session for tez and llap > > > Key: HIVE-11923 > URL: https://issues.apache.org/jira/browse/HIVE-11923 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 1.3.0, 2.0.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-11923.1.txt, HIVE-11923.2.branchllap.txt, > HIVE-11923.2.patch, HIVE-11923.2.txt, HIVE-11923.2.txt, > HIVE-11923.branch-1.txt > > > Launching a new session - AM and containers for each test adds unnecessary > overheads. Running via a single session should reduce the run time > significantly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11923) allow qtests to run via a single client session for tez and llap
[ https://issues.apache.org/jira/browse/HIVE-11923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949377#comment-14949377 ] Sergey Shelukhin commented on HIVE-11923: - Will update patch soonish > allow qtests to run via a single client session for tez and llap > > > Key: HIVE-11923 > URL: https://issues.apache.org/jira/browse/HIVE-11923 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 1.3.0, 2.0.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-11923.1.txt, HIVE-11923.2.branchllap.txt, > HIVE-11923.2.patch, HIVE-11923.2.txt, HIVE-11923.2.txt, > HIVE-11923.branch-1.txt > > > Launching a new session - AM and containers for each test adds unnecessary > overheads. Running via a single session should reduce the run time > significantly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11923) allow qtests to run via a single client session for tez and llap
[
https://issues.apache.org/jira/browse/HIVE-11923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14951875#comment-14951875
]
Hive QA commented on HIVE-11923:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12765694/HIVE-11923.03.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 9659 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_scriptfile1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_stats_counter
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_stats_counter_partitioned
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_temp_table_external
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_truncate_column_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_uber_reduce
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_scriptfile1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_scriptfile1
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5596/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5596/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5596/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12765694 - PreCommit-HIVE-TRUNK-Build
> allow qtests to run via a single client session for tez and llap
>
>
> Key: HIVE-11923
> URL: https://issues.apache.org/jira/browse/HIVE-11923
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-11923.03.patch, HIVE-11923.1.txt,
> HIVE-11923.2.branchllap.txt, HIVE-11923.2.patch, HIVE-11923.2.txt,
> HIVE-11923.2.txt, HIVE-11923.branch-1.txt
>
>
> Launching a new session - AM and containers for each test adds unnecessary
> overheads. Running via a single session should reduce the run time
> significantly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11923) allow qtests to run via a single client session for tez and llap
[
https://issues.apache.org/jira/browse/HIVE-11923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955041#comment-14955041
]
Hive QA commented on HIVE-11923:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12766239/HIVE-11923.04.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 9660 tests executed
*Failed tests:*
{noformat}
TestCliDriver-ppd_outer_join2.q-groupby3_map_skew.q-merge_dynamic_partition2.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5628/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5628/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5628/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12766239 - PreCommit-HIVE-TRUNK-Build
> allow qtests to run via a single client session for tez and llap
>
>
> Key: HIVE-11923
> URL: https://issues.apache.org/jira/browse/HIVE-11923
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-11923.03.patch, HIVE-11923.04.patch,
> HIVE-11923.1.txt, HIVE-11923.2.branchllap.txt, HIVE-11923.2.patch,
> HIVE-11923.2.txt, HIVE-11923.2.txt, HIVE-11923.branch-1.txt
>
>
> Launching a new session - AM and containers for each test adds unnecessary
> overheads. Running via a single session should reduce the run time
> significantly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11923) allow qtests to run via a single client session for tez and llap
[
https://issues.apache.org/jira/browse/HIVE-11923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955455#comment-14955455
]
Sergey Shelukhin commented on HIVE-11923:
-
[~prasanth_j] [~hagleitn] can you take a look at the updated patch?
The bad instance failed due to some spark(?) download nonsense.
{noformat}
[exec] + /bin/pwd
[exec] + BASE_DIR=./target
[exec] + HIVE_ROOT=./target/../../../
[exec] + DOWNLOAD_DIR=./../thirdparty
[exec] + mkdir -p ./../thirdparty
[exec]
/home/hiveptest/54.147.252.66-hiveptest-1/apache-github-source-source/itests/hive-unit
[exec] + download
http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.4.0-bin-hadoop2-without-hive.tgz
spark
[exec] +
url=http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.4.0-bin-hadoop2-without-hive.tgz
[exec] + finalName=spark
[exec] ++ basename
http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.4.0-bin-hadoop2-without-hive.tgz
[exec] + tarName=spark-1.4.0-bin-hadoop2-without-hive.tgz
[exec] + rm -rf ./target/spark
[exec] + [[ ! -f ./../thirdparty/spark-1.4.0-bin-hadoop2-without-hive.tgz
]]
[exec] + tar -zxf ./../thirdparty/spark-1.4.0-bin-hadoop2-without-hive.tgz
-C ./target
[exec] + mv ./target/spark-1.4.0-bin-hadoop2-without-hive ./target/spark
[exec] + cp -f ./target/../../..//data/conf/spark/log4j2.xml
./target/spark/conf/
[exec] + sed '/package /d'
/home/hiveptest/54.147.252.66-hiveptest-1/apache-github-source-source/itests/../contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleAdd.java
[exec] + javac -cp
/home/hiveptest/54.147.252.66-hiveptest-1/maven/org/apache/hive/hive-exec/2.0.0-SNAPSHOT/hive-exec-2.0.0-SNAPSHOT.jar
/tmp/UDFExampleAdd.java -d /tmp
[exec] + jar -cf /tmp/udfexampleadd-1.0.jar -C /tmp UDFExampleAdd.class
[exec] java.util.zip.ZipException: attempt to write past end of STORED
entry
[exec] at java.util.zip.ZipOutputStream.write(ZipOutputStream.java:319)
[exec] at sun.tools.jar.Main.copy(Main.java:779)
[exec] at sun.tools.jar.Main.copy(Main.java:793)
[exec] at sun.tools.jar.Main.addFile(Main.java:740)
[exec] at sun.tools.jar.Main.create(Main.java:491)
[exec] at sun.tools.jar.Main.run(Main.java:201)
[exec] at sun.tools.jar.Main.main(Main.java:1177)
{noformat}
> allow qtests to run via a single client session for tez and llap
>
>
> Key: HIVE-11923
> URL: https://issues.apache.org/jira/browse/HIVE-11923
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-11923.03.patch, HIVE-11923.04.patch,
> HIVE-11923.1.txt, HIVE-11923.2.branchllap.txt, HIVE-11923.2.patch,
> HIVE-11923.2.txt, HIVE-11923.2.txt, HIVE-11923.branch-1.txt
>
>
> Launching a new session - AM and containers for each test adds unnecessary
> overheads. Running via a single session should reduce the run time
> significantly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11923) allow qtests to run via a single client session for tez and llap
[ https://issues.apache.org/jira/browse/HIVE-11923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955494#comment-14955494 ] Prasanth Jayachandran commented on HIVE-11923: -- +1 for the workaround patch. > allow qtests to run via a single client session for tez and llap > > > Key: HIVE-11923 > URL: https://issues.apache.org/jira/browse/HIVE-11923 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 1.3.0, 2.0.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-11923.03.patch, HIVE-11923.04.patch, > HIVE-11923.1.txt, HIVE-11923.2.branchllap.txt, HIVE-11923.2.patch, > HIVE-11923.2.txt, HIVE-11923.2.txt, HIVE-11923.branch-1.txt > > > Launching a new session - AM and containers for each test adds unnecessary > overheads. Running via a single session should reduce the run time > significantly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
