[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15504870#comment-15504870 ] Sergey Shelukhin commented on HIVE-14651: - lgtm +1 > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, > HIVE-14651.03.patch, HIVE-14651.04.patch, HIVE-14651.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15504277#comment-15504277 ] Prasanth Jayachandran commented on HIVE-14651: -- +1 > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, > HIVE-14651.03.patch, HIVE-14651.04.patch, HIVE-14651.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15504276#comment-15504276 ] Prasanth Jayachandran commented on HIVE-14651: -- That's the one. Makes sense. > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, > HIVE-14651.03.patch, HIVE-14651.04.patch, HIVE-14651.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15504272#comment-15504272 ] Siddharth Seth commented on HIVE-14651: --- Which comment are you referring to? (The one which says "Force fs to file://, setup staging dir?) The staging dir is setup within the Hive directories, otherwise Tez takes care of creating the dirs based on configuration. Once local mode works properly, these comments will get resolved (I'd like to allow local mode to work with either file:// or hdfs. Tez does not support hdfs with local mode yet) > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, > HIVE-14651.03.patch, HIVE-14651.04.patch, HIVE-14651.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15504223#comment-15504223 ] Prasanth Jayachandran commented on HIVE-14651: -- [~sseth] regarding your comments in the code. how does it work without setting up staging dir? Is there a default that tez AM sets up? > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, > HIVE-14651.03.patch, HIVE-14651.04.patch, HIVE-14651.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15504202#comment-15504202 ] Siddharth Seth commented on HIVE-14651: --- [~prasanth_j], [~sershe] - the test results look good now. Any other comments? > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, > HIVE-14651.03.patch, HIVE-14651.04.patch, HIVE-14651.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15498499#comment-15498499 ] Hive QA commented on HIVE-14651: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12828917/HIVE-14651.05.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10561 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char] org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning] org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1221/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1221/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1221/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12828917 - PreCommit-HIVE-MASTER-Build > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, > HIVE-14651.03.patch, HIVE-14651.04.patch, HIVE-14651.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15450931#comment-15450931 ] Siddharth Seth commented on HIVE-14651: --- Hmm. I think you're right. Somehow missed that altogether. Will investigate some more. Really need to get the tests to be reliable - or at least close to reliable. Local cluster - run everything in a single process. For LLAP - this would mean avoiding AM launches. For MiniTez, avoid AM launches and container launches. Everything runs inline. > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, > HIVE-14651.03.patch, HIVE-14651.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15450464#comment-15450464 ] Sergey Shelukhin commented on HIVE-14651: - Well, complete failure of TestMiniSparkOnYarnCliDriver is suspicious. Some small comments on RB. Would be nice to describe what is local cluster, etc. > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, > HIVE-14651.03.patch, HIVE-14651.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15450383#comment-15450383 ] Siddharth Seth commented on HIVE-14651: --- I don't think the current set of failures are related. Have seen all of these fail before. Fixed hybridgraceHashjoin1.q from the last run. Anyway, RB link is here: https://reviews.apache.org/r/51543/ > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, > HIVE-14651.03.patch, HIVE-14651.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15450256#comment-15450256 ] Sergey Shelukhin commented on HIVE-14651: - Failures may be related. Is it possible to post an RB? > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, > HIVE-14651.03.patch, HIVE-14651.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15450250#comment-15450250 ] Hive QA commented on HIVE-14651: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12826225/HIVE-14651.04.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10407 tests executed *Failed tests:* {noformat} TestClearDanglingScratchDir - did not produce a TEST-*.xml file TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char] org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1045/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1045/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1045/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12826225 - PreCommit-HIVE-MASTER-Build > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, > HIVE-14651.03.patch, HIVE-14651.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15441764#comment-15441764 ] Hive QA commented on HIVE-14651: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12825728/HIVE-14651.03.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10405 tests executed *Failed tests:* {noformat} TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char] org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning] org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[hybridgrace_hashjoin_1] org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1021/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1021/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1021/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12825728 - PreCommit-HIVE-MASTER-Build > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, > HIVE-14651.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440097#comment-15440097 ] Prasanth Jayachandran commented on HIVE-14651: -- nit: LOG.error instead? {code} try { fs.mkdirs(userInstallPath); } catch (IOException e) { LOG.info("Failed to create path={}. Continuing. Exception message={}", warehousePath, e.getMessage()); } {code} There are 2 other places as well. Other that that lgtm, +1 > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, > HIVE-14651.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440077#comment-15440077 ] Siddharth Seth commented on HIVE-14651: --- It should not be. Will upload a patch with this removed. > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440058#comment-15440058 ] Prasanth Jayachandran commented on HIVE-14651: -- Mostly looks good. Why is llap.io disabled? {code} if (isLlap) { conf.setBoolean("hive.llap.io.enabled", false); } {code} > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439586#comment-15439586 ] Hive QA commented on HIVE-14651: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12825619/HIVE-14651.02.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10404 tests executed *Failed tests:* {noformat} TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char] org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning] org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_join30] org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1007/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1007/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1007/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12825619 - PreCommit-HIVE-MASTER-Build > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)